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Abstract 

Let r be a countable abelian semigroup and A be a countable abelian group 
satisfying a certain finiteness condition. Suppose that a group G acts on a 
(r x _4)-graded Lie superalgebra £ = a ) e rx^4 £(a,a) by Lie superalgebra 
automorphisms preserving the (r x „4)-gradation. In this paper, we show that 
the Euler-Poincare principle yields the generalized denominator identity for £ 
and derive a closed form formula for the supertraces str(^|£/ aj0 )) for all g € G, 
(a, a) £ r x A. We discuss the applications of our supertrace formula to various 
classes of infinite dimensional Lie superalgebras such as free Lie superalgebras and 
generalized Kac-Moody superalgebras. In particular, we determine the decompo- 
sition of free Lie superalgebras into a direct sum of irreducible GL(n) x GL(k)- 
modules, and the supertraces of the Monstrous Lie superalgebras with group 
actions. Finally, we prove that the generalized characters of Verma modules and 
the irreducible highest weight modules over a generalized Kac-Moody superal- 
gebra g corresponding to the Dynkin diagram automorphism a are the same as 
the usual characters of Verma modules and irreducible highest weight modules 
over the orbit Lie superalgebra g = q{o) determined by a. 
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1 Introduction 



In the past three decades, the theory of infinite dimensional Lie algebras and their 
representations has been the focus of extensive research activities due to its rich and 
significant applications to many areas of mathematics and mathematical physics. The 
most well-known examples are Kac-Moody algebras (|[27||, and generalized Kac- 



Moody algebras (||), and a lot of exciting new discoveries have been made using the 
language of these infinite dimensional Lie algebras. For example, the Macdonald iden- 
tities ([£5|) were shown to be equivalent to the denominator identities of affine Kac- 
Moody algebras (||28|), and the Moonshine Conjecture was proved by investigating the 
structure of a special kind of generalized Kac-Moody algebra called the Monster Lie 
algebra (0). 

On the other hand, since 1970's, the Lie superalgebras and their representations 
have emerged naturally as the fundamental algebraic structure behind several areas of 
mathematical physics. In V. G. Kac gave a comprehensive presentation of the 
mathematical theory of Lie superalgebras, and obtained an important classification 
theorem for finite dimensional simple Lie superalgebras over algebraically closed fields 
of characteristic zero (see also The theories of Kac-Moody algebras and gener- 

alized Kac-Moody algebras have been extended to those of Kac-Moody superalgebras 
(p0|) and generalized Kac-Moody superalgebras (fl47|, [fyj) following the outline given in 
|3l|| . In [Q and f35f , V. G. Kac and M. Wakimoto developed the representation theory 
of affine Kac-Moody superalgebras, and demonstrated some interesting applications of 
affine Kac-Moody superalgebras to number theory. 

In ||37|| , S.-J. Kang derived a closed form root multiplicity formula for generalized 
Kac-Moody algebras, and by applying his formula to the Monster Lie algebra, he ob- 
tained some interesting recursive relations among the coefficients of the elliptic modular 
function j. The main idea of |37J is that the Euler-Poincare principle for the Lie alge- 
bra homology can be interpreted as the denominator identity for Kac-Moody algebras 
or generalized Kac-Moody algebras, and once we are given the denominator identity, 
we can derive a root multiplicity formula. This idea has been extended to the general 
infinite dimensional graded Lie algebras to yield a closed form dimension formula for 
the homogeneous subspaces (f42|), and a lot of interesting applications of our formula 
to various classes of infinite dimensional Lie algebras have been investigated (see also 
and [0). In |j32| , V. G. Kac and S.-J. Kang exploited this idea further to derive a 



closed form trace formula for graded Lie algebras with group actions. In particular, by 
applying their trace formula to the Monster simple group acting on the Monster Lie 
algebra, they obtained a family of interesting recursive relations among the coefficients 
of the Thompson series. 

The Thompson series are examples of replicable functions which are normalized q- 
series satisfying certain functional equations called the replication formulae. In |p[0|| , 
S.-J. Kang showed that, given the denominator identity for graded Lie superalgebras, 
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one can derive a closed form superdimension formula for the homogeneous subspaces, 
which can be considered as a unifying method of studying the structure of infinite 
dimensional Lie superalgebras. He discussed a lot of interesting applications of our 
superdimension formula to various classes of infinite dimensional Lie superalgebras 
such as free Lie superalgebras, generalized Kac-Moody superalgebras, and Monstrous 
Lie superalgebras. Furthermore, he characterized the replicable functions in terms of 
certain product identities, and determined the root multiplicities of the Monstrous Lie 
superalgebras associated with replicable functions. 



In this paper, we combine the main ideas of |32| and |40| to derive a closed form 
supertrace formula for graded Lie superalgebras with group actions. More precisely, let 
T be a countable (ususally infinite) abelian semigroup and A be a countable (usually 
finite) abelian group such that every element (a, a) £ T x A can be written as a sum of 
elements in T x A in only finitely many ways. Suppose that a group G acts on a graded 
Lie superalgebra £ = 0/ a ) grx _4 £(a,a) by Lie superalgebra automorphisms preserving 
the (r x ^4)-gradation. Then, in Section 2, we show that the Euler-Poincare principle 
for the Lie superalgebra homology yields a product identity 



J] exp(-f^istr(/|£ (a , a) )^ a )) 

(a,a)erxA V fc=i / 



= 1 - Mg\H(£) ia , a) )E^ a \ 

(a,a)eFxA 

which is called the generalized denominator identity for g G G. By taking the logarithm 
of both sides of the generalized denominator identity and using Mobius inversion, we 
derive a closed form formula for the supertraces str(g|£( aja )) = ip(a)tr(g\£^ a )) for all 
g E G, (a, a) E T x A (Theorem O): 



str(#|£ (Qia) ) = \v{d)W g d(T,b), 



d>0 
(a,a)=d(T,b) 

where ip(a) is the sign map defined on A and W 9 (t, b) is the Witt partition function 
defined by (|2~T6| ). 

In Section 3, we present the standard homology theory for Lie superalgebras. That 
is, we show that there exists a super- analogue of Koszul's complex for Lie superalgebras 
(Proposition |3.1|). The &;-th homology module H k (£, V) of the Lie superalgebra £ with 
coefficients in V is defined by 

H k (£, V) = To^(VX) = H k (V®u M), 

where M = (M&., d^) is the super-analogue of Koszul's complex given by ( |3.1| ) and ( |3.2| ), 
and U = U(£) denotes the universal enveloping algebra of £. In particular, if £(V) is 
the free Lie superalgebra generated by a superspace V, then we have Hi(2{V)) = V 



3 



and Hk(£(V)) = for all k > 2. Even though our presentation is a straightforward 
generalization of the homology theory of Lie algebras (cf. [pTOfl , [0), we decided to 
include the detailed treatment here, for we could not find it in any other literature. 

Section 4 and Section 5 are devoted to the applications of our supertrace formula 
to free Lie super algebras. In Section 4, we generalize the classical Witt formula for free 
Lie algebras to obtain a closed form supertrace formula for free Lie superalgebras with 
group actions. As an application, we compare the structure of the free Lie algebras 
and free Lie superalgebras generated by the same group representations (Proposition 
|4.2|) . In Section 5, we consider the free Lie superalgebra £(V) generated by the natural 
(fc + /)-dimensional representation V of the general linear Lie superalgebra gl(k, I), and 
determine the decomposition of £(V) into a direct sum of irreducible GL(k) x GL{1)- 
modules (Proposition |0| and Proposition |5.4j ). The Hook Schur functions and the 
Littlewood- Richards on coefficients are the key ingredients of the decomposition. 

In Section 6 and Section 7, we give a conjectural Kostant-type formula for the 
homology of the negative part of generalized Kac-Moody superalgebras, and derive a 
closed form supertrace formula for generalized Kac-Moody superalgebras with group 
actions preserving the root space decomposition (Theorem |6.5| ). Moreover, we apply 
our supertrace formula to determine the supertraces of the Monstrous Lie superal- 
gebras with group actions preserving the root space decomposition (Proposition [TT] 
and Corollary |TT^) . We believe that the product identities characterizing the replica- 
ble functions (j|(J) can be understood most naturally as the generalized denominator 
identities of some Mosntrous Lie superalgebra with a suitable group action. 

In Section 8 and Section 9, we prove that the generalized characters of Verma 
modules and irreducible highest weight modules over a generalized Kac-Moody super- 
algebra q corresponding to the Dynkin diagram automorphism a are the same as the 
usual characters of Verma modules and irreducible highest weight modules over the 
orbit Lie superalgebra q = fj(cr) determined by o (Theorem |9.3| and Corollary |9.4|) . 
This fact was first proved in |L3| for symmetrizable Kac-Moody algebras under the 
linking condition in solving the fixed point problem in conformal field theory and was 



generalized to generalized Kac-Moody algebras without the linking condition in JT4 



In this paper, we also show that the generalized denominator identity for a generalized 
Kac-Moody superalgebra q for the Dynkin diagram automorphism a is the same as 
the usual denominator identity for the orbit Lie superalgebra g = g(a) (Theorem |9.5|) . 
Our argument for generalized Kac-Moody superalgebras is similar to those in [13| and 
14}| . We observe that the generalized character of a Verma module can be written as 



the product part of a generalized denominator identity and derive an explicit formula 
for the traces of a on each invariant subspaces (which are not the same as root spaces 
in general) of q in terms of the dimensions of some orbit Lie superalgebras. Therefore, 
we obtain a dimension formula for each homogeneous subspaces of the invariant sub- 
algebra, which is also a generalized Kac-Moody superalgebra with a natural grading 
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arising from a (Proposition |9.6|) . 



Acknowledgments. The authors would like to express their sincere gratitude to 
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Graded Lie Superalgebras and Supertrace For- 
mula 



We recall some basic facts about Lie superalgebras (P5|, P0| , |53fl ). Let A be a count- 
able abelian group and suppose we have a bimultiplicative map 9 : A x A — ► C x 
satisfying 

9(a + b, c) = 6(a,c)9(b,c), 
(2.1) 9{a, b + c) = 9{a,b)9{a,c), 

9(a, b)8(b, a) = 1 for all a,b G A. 

In particular, we have 9(a, a) = ±1 for all a G A. The map 9 satisfying the condition 

(2.1) is called a coloring map on A. Define a map ip : A — ► {±1} by ip(a) = 9(a,a). 
Then ip is a well-defined group homomorphism called the sign map on A with respect 
to 9. Let Ao = { a G A \ if) (a) = 1 } and Ai = { a G A \ if) (a) = —1 }. Then we have a 
decomposition A = Ao U A\ and the elements of A in Aq (resp. A\) are called even 
(resp. odd). 

A 9 -colored superspace is a pair (V, 9), where V = © ae _4 V a is an ^4-graded vector 
space and 9 : A x A ^ <C X is a coloring map on A. The elements of V a are called 
even (resp. odd) if ip(a) = 1 (resp. ip(a) = —1). For each a G A, we define the 
superdimension of V a to be 

(2.2) sdimK = ^(a)dimK- 

Similarly, we define a 9 -colored superalgebra to be a pair (U, 9), where U = © ag _4 U a 
is an ^4-graded associative algebra (i.e., UJJ\, C U a+ b for all a, b G ^4) and 6 1 : A x ^4 — > 
C x is a coloring map on A. The direct sum of ^4-graded superalgebras is defined in 
the usual way, but, for ^-colored superalgebras U = © ae _4 U a and U' = © a / e _4 U' a , we 
define the tensor product of U and U' to be the ^-colored superspace U ®U' with the 
natural ^4-gradation 

(U ® f/')a+a' = Span{w ® «'| u G C/ a , u 7 G E/^} 
and the multiplication given by 

(u ® m')(w ® u') = 9(a', b)(uv ® mV) 
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for ueU a ,ve U b , u' G U' a ,, v' G U'y, a, a', b, b' G A. 

Definition 2.1 A 9-colored Lie superalgebra is a ^-colored superspace £ = © ae _4 £ a 
together with a bilinear operation [ , ] : £ x £ — > £ satisfying 

[£ a , £b] c 

(2.3) [x,y] = -0(a,&)[y,x], 

[x, [y, A\ = [i x : y],A + h ) [v, i x , A\ 

for all x G £ a , y G £&, z6£ and a,b £ A. 
Let £ = © 

aeA) ^ a an< ^ ^ ~~ ©feeA £&■ Then we have a decomposition £ — 
£o © £i, and the homogeneous elements of £o (resp. £i) are called even (resp. odd). 

The universal enveloping algebra of a ^-colored Lie superalgebra £ = © ae _4 £ a is 
the pair (£/(£), t)> where £/(£) is a 6*-colored superalgebra and i : £ — > E/(£) is a linear 
mapping satisfying 

l([x, y\) = i{x)i{y) - 9(a, b)i(y)i(x) for x G £ OJ y G £ fe 

such that for any ^-colored superalgebra C/ = aeA U a and a linear mapping j : £ — > U 
satisfying 

j'Cfo y]) = j( x )j(y) - #K b)j(y)j(x) for a; G £ , y G £&, 
there exits a unique homomorphism ip : t/(£) — > ?7 of ^-colored superalgebras satisfying 

1pOL = j. 

The uniqueness of the universal enveloping algebra [/ (£) can be proved in the usual 
way For the existence, consider the tensor algebra T(£) = ©fcl £® fc of £ and let 1Z 
be the ideal of T(£) generated by the elements of the form 

[x, y] — x <g> y + 6>(a, b)y ® x (x G £ a , y G £&, a, 6 G .A). 

Then the quotient algebra ?/(£) = T(2)/1Z together with the natural mapping i : £ — ► 
[/(£) is the universal enveloping algebra of £. 

For the structure of the universal enveloping algebra U(£) of £, the following version 
of Poincare-Birkhoff-Witt theorem is well-known. 



Theorem 2.2 (§, p|, pf) 



Let A be a countable abelian group with a coloring map 9 : A x A — > C x and 
Ze£ £ = © ae _4£ a a 9-colored Lie superalgebra. Suppose X = {x a \ a G A} (resp. 
Y = {yp\ P e ^}) ^ s a homogeneous basis of the subspace £ = ® ae ^ £a (resp. 
£i = © be- 4 £& ) ■ T/ien £/ie elements of the form 

(2.4) x ai x a2 • • • x ah yfaVfo ■■■ypi with a x < ■ • • < a k , /3i < • • • < fa 

together with 1 form a basis of the universal enveloping algebra U (£) of £. □ 
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Remark. The theory of colored Lie superalgebras can be reduced to the theory of 
ordinary Z 2 -graded Lie superalgebras by redefining the superbracket in a suitable way. 
More precisely, let £=(£,[ , ]) be a ^-colored Lie superalgebra with a coloring map 
9 : A x A — > C x and let X = {xi\ % = 1, 2, ■ ■ • } be an ordered set of generators of A. 
We define the bimultiplicative maps 5 : A x A — > C x and r : A x A — > C x by 

6(a,b) = (-l)^ (o)v,(6) 0(M) 

and 

r(a,b) = Y[8(xi,Xj) riSj , 

i<3 

where a = J2 r i x i> b = ^2 s i x i £ A (fj, Sj G Z). We now introduce a new bracket 
[ , } T : £ x £ — > £ defined by 

[x,y} T = r(a,b)[x,y] 

for x G £ a , y G £fe. Then £ T = (£, [ , ] r ) becomes an ordinary Z 2 -graded Lie su- 
peralgebra. Furthermore, there is a 1-1 correspondence between the set of ^4-graded 
representations of the ^-colored Lie superalgebra £=(£,[, ]) and the set of ^4-graded 
representations of the ordinary Lie superalgebra £ T = (£, [ , ] r ) (cf. |)0|]. [54|). 



Let A be a countable (usually finite) abelian group with a coloring map 9, and let 
T be a countable (usually infinite) abelian semigroup such that every element (a, a) G 
Fx A can be written as a sum of elements in T x A in only finitely many ways. Consider 
a (r x *4)-graded ^-colored superspace V = ®/ Q a ) grx _4 V(a,a) with dimV^a) < oo for 
all (a, a) G T x A. Suppose that a group G acts on V preserving the (r x .4) -gradation. 
We define the generalized character or the graded trace of V for g G G by 

(2.5) ch s ^= Yl ^(9\V(a,a))e {a ' a \ 

(a,a)eFxA 

where e^'^ are the basis elements of the semi-group algebra C[T x A] with the multi- 
plication given by e ( a > a ) e W> b ) = e {»+p,a+b) _ 

On the other hand, we define the supertrace of the homogeneous subspace V^ a ) for 
geGby 

(2.6) BtT(g\V M ) = i/>(a)tr(g\V M ), 

and introduce another basis elements of C[r x .4] by setting E^'^ = ?/>(a)e (a ' a - ) . Then 
it is easy to verify that E^^E^^ = E^ a+l3 ' a+b \ We now define the generalized super- 
character or the graded supertrace of V for g G G by 

(2.7) sch g V= Yl M9\V (a ,a))E^ a \ 

Note that sch g V is obtained from ch 9 V^ by replacing tr(g\V( a>a }) = ^(a)sti(g\ V( Qja )) and 
e {a,a) _ ^( a )£( Q . a ). Since ip{a) 2 = 1, we have ch g V = sch g V. 
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Let £ = ®( a s ) e r x ^ ^(a,a) be a ^-colored (T x ^4)-graded Lie superalgebra with 
dimH( aa ) < oo for all (a,fl) 6 Txi and suppose that a group G acts on £ by Lie 
superalgebra automorphisms preserving the V x ^4-gradation. In this work, using the 
Euler-Poincare principle, which is equivalent to the generalized denominator identity, 
we will derive a closed form formula for the supertraces str(g|£( Q a )) for all g G G, 
(a, a) e r x A. 

Let C be the trivial one dimensional £-module. The homology modules Hk(2) = 
iffe(£, C) are determined from the following standard complex: 

(2.8) • • • - CftOC) ^ C fc _i(£) ^ • • • - d(£) ^ C (£) - 0, 

where Cfc(£) are defined by 

= A p (£ ) ® S 9 (£i) 

p+q=fc 

and the differentials d k : Cfe(£) — > Cj._x(£) are given by 
4((xi A • ■• Ai p ) ® (j/i- • -j/,)) 

= ^ (-l) s+ *([x s ,x t ] Aii A---Ax fl A---Afi A • • • Aar p ) ® (y x • • -y ff ) 

l<s<t<p 

v q 

+ Z^ -1 )^ 1 a • • • A f; A • • • A x p ) ® ([x„ yt\yi ■■■yf-y q ) 

8=1 t=l 

- Z ([ys^yt] A ■ ■ ■ Axp) (g) (yi- ■ -y s - ■ -y t - ■ -y q ) 

l<s<t<q 

for k > 2, Xi G £o, G £i and d\ — (cf. [flO|l , jOJ). In Section 3, we will give a more 
detailed treatment of the homology of £. Since the spaces Cfc(£) and the homology 
modules H}.{£) inherit the (T x ^4)-gradation from that of £, the action of G and 
the generalized supercharacters of Ck{£) and -£/&(£) are well-defined. Hence by the 
Euler-Poincare principle, we obtain 



(2.9) ^(-l) fc sch s C fc (£) = ^(-l) fc sch s # fc (£) for all g G G. 

k=0 k=0 

Let 

oo 

C(£) = ^(-l) fc C fe (£) = c e L © C 2 (£) e • • • , 

fc=0 
oo 

A(£ ) = ^(-l) fc A fc (£ ) = C e £ © A 2 (£ ) © • • • , 

fc=0 
oo 

5(£0 = ^(-l)^ fc (£i) = C © £x © 5 2 (£i) © • • • , 

fc=0 

oo 

#(£) = ^(-l)* +1 # fc (£) = i?!(£) © F 2 (£) © F 3 (£) © • • • 



(2.10) 



fc=i 
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the alternating direct sum of superspaces. Then it is easy to see that C(£) = A(£ ) <8> 
5(£i). 

We recall the very basic theory of symmetric functions. Let x%, ■ ■ ■ , x k be indeter- 
minates and for each n G N, define 



p n (xi,--- ,x k ) = J^x", 
i=i 

[2.11) K(xi, • ■ ■ ,Xk) = ^2 Xil ' 

ii<---<i„ 



ii<— <i„ 

which are called the nth power sum, the n£/i completely symmetric function, and the 
nth elementary symmetric function, respectively. Note that the elementary symmetric 
function e n is defined only for n < k. To compute the generalized supercharacters of 
A(£o) and 5'(£i), we recall the following fundamental lemma in the theory of symmetric 
functions: 



Lemma 2.3 ([§§]) 



(2.12) exp(£^)=$>t 



r>l r / n>0 ^n>0 



E„>o(-i) n ^ n ' 

□ 



Let A be an n x n complex matrix with eigenvalues x\, - ■ ■ , x n . By Lemma |2l| we 
have 



exp 



V 



;2.i3) 



,r>l 



exp I - ^ tr(A r 



J2^( A \s m (c n ))t m , 

m>0 

^(-l) m tr(A|A m (C n ))t r ' 



m>0 



Let t = (i(cn, a))(a,a)erxA be a sequence of nonnegative integers indexed by T x A 
with only finitely many nonzero terms, and set \t\ = J^t(a,a). Since the kth exterior 
power A fc (£ ) is decomposed as 

/ 

A fc (£o) = (g) A t(a ' a) (%*,a) 



\t\=k y(a,a)Grx^o 



we have 



ch 9 A(£ ) = ^(-l) fe ch 9 A fc (£ 



fc=0 



dim£ (aja) 

) 



II E (-l) m tr(g\A m (^ a , a) ))e m ^ 

0,a)erxA> \ m =o 



n E(- i ) mtr ^i Am (^))) emKa 



(a,a)eTxAo \m>0 



(a,a)er 

for all g e G. Similarly, we have 



II exp[-£ltr(/|£ (Q , a) )e^) > ) 

a,a)€rxA V r=l J 



ch^fix) = £(-l)*ch fl S fc (,d) 



It follows that 



fc=0 

n (d-^^i^w)^ 

(a,o)erxA \m>0 

II (^tr(^|^(% i6) ))(-e^))- 

(a,o)erx^i \m>0 

II exp(f t^tr(/|£ (/3 , b) )e^) 

(a,a)erx^i \r=l 



ch g C(£) = ^(-l) fe ch g C fe (£) = ch 9 A(£ ) • ch fl S , (£i) 
= II exp(-^itr(/|£ (a , a) )e^)j 

(a,a)eTxAa \ r=l J 



x II exp(^^tr(/| % , 6) )e^) 

(P,b)eTxAi \r=l 



Replacing E^ a ' a ^ = ■?/>(a)e (a ' a ) and str(g|£( QjCl )) = ^(a)tr(^|£( aja )), the above identity 
yields 

sch 9 C(£) = J] exp f-f^istr(/|£ (a)0) )^( Q '°) ) . 

(o,a)erxyl V r=l / 

Hence, by the Euler-Poincare principle, we obtain the generalized denominator identity 
for the ^-colored graded Lie superalgebra £ = ©(„ a ) e rx>i^M : 

(2.14) J] ex P (-f;^tr(^|£(a,a))^ r(Q ' a) J = 1 -sch g tf(£). 

(a,a)eTxA V r=l / 
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When g = 1, the identity element of G, we get the denominator identity for the Lie 
super algebra £ 



Y[ (1 - £ (Q ' a) ) sdim£ (<-> = 1 - sc h#(£). 



rx.4 



Let P(r x ^4) = {(a, a) G T x ,A| str(#|F(£) (cMI )) 7^ 0} and { (c^, a,) |i, j = 
1, 2, 3, ■ • • } be an enumeration of P(T x ^4). For (r, 6) G T x A, set 

(2.15) T(t, 6) = { s = (sij) I Sij > 0, ^ %(o:i, %) = (r,b) }, 
the set of all partitions of (t, 6) into a sum of («j, a,,')'s, and define 

(2.16) W g (r,b)= J2 ^^n^^l^^K^))^- 

s&T(r,b) ' ' i,j 

We will call W g (r, b) the Witt partition function for g G G. In the next theorem, using 
the generalized denominator identity (2.14), we derive a closed form formula for the 
supertraces str(g|£( aja )) ((a, a) G T x 4) in terms of the Witt partition functions. 

Theorem 2.4 For g G G and (a, a) e T x we have 

(2.17) sfr<0|£ (Q , a) ) = E ^(d)W^(r,6). 

d>0 
(a,a)=d(r,b) 



Proof. By the generalized denominator idenity (|2.14 ), we have 



eX p( E £^tr(^|£(a,a))^ r(Q ' a) I 
\(a,a)erx4 r=l J 



l-ES=iStr(^|i?(£) (ail% .))^^)- 



Taking the logarithm and using the formal power series log(l — x) = — ^2™ =1 kx k , we 
obtain 



r 

(a,a)GFxA r=l 



00 

E E- str ^ r i^))^ (Q ' a) 

r=l 

00 

-io g (i - ^(gmty^E^y 1 

00 1 / 00 \ m 

m=l \m=1 / 
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e^ e %f#n str ^i^( £ )K^)) si ^ E ^ (ai ' ai) 



E Sij=m 



E (e (£ n J 3 ., 1)! n^tol g ( £ )«..^))" J ) ^ 

(r,b)erx^ \seT(r,6) 11 4i ' / 



(r,6)erx^ 

It follows that 

W g (r,b)= lstr(9 k \Z(a, a) ). 

k>0 
(r,b)=k(a,a) 

Hence, by Mobius inversion, we obtain 

Btr^l £(«,«)) = E ^M^W^r,^. 

d(r,b)=(o,a) 



□ 



Remark. Note that our supertrace formula is a generalization of the trace formula for 
graded Lie algebras obtained in [Q. Also, when g = 1, our supertrace formula yields 
the superdimension formula for homogeneous subspaces of graded Lie superalgebras 

(cf. H). 



3 Homology of Lie Superalgebras 

The homology group of a Lie algebra is defined as the torsion group of its universal 
enveloping algebra and characterized by Koszul's complex. Similarly, we define the 
homology group of a graded Lie superalgebra as the torsion group of its universal en- 



veloping algebra viewed as a supplmented algebra (cf. |Kj, |T5|]). In this section, we 
will show that there is a super-analogue of Koszul's complex for a graded Lie superal- 
gebra and characterize its homology group. Our argument follows the framework given 
in 



HJ and p|. 



Let £ = £ © £i be a Lie superalgebra and U = U (£) be its universal enveloping 
algebra. For each k > 0, define 

(3.1) C k = C,(£) = A p (£ ) ® ^(£x). 

p+q=k 
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Consider the following chain complex (M fc ,<9^) (k > — 1), where 

\u® c C k {£) iffc>0, 

lVl k — \ 

(C if A; = — 1, 

and the differentials d k : — ► Mfc_i are given by 
(3.2) 

<9 fc (w <g) (xi A • • • A x p ) <g> (j/i • • • j/ ff )) 

= ^ (-l) s+ *w® ([x s ,x t ] Axi A---Ax 8 A---Ax t A---Ax p )®(y 1 ---y q ) 

l<s<t<p 

p g 

+ ^2 X^ -1 ) 5 " ® (^i a • • • a x s a • • • a x p ) <g> ([x s , y t ]yi •• -yf-yq) 

s=l t=l 

- M Axi A ■ • • A Xp) <g> (y 1 ■ ■ -y s - ■ -y t ■ ■ -y q ) 

l<s<t<q 
V 

+ ^2(-l) s+1 (u ■ x s ) <g> (xi A • • • A x s A • • • A x p ) <g> (y x • • -y q ) 

s=l 

+ (-1 ) p J^(w • j/ t ) <g> ( Xl A ■ ■ ■ A x p ) <g> (yi • • • y t ■ ■ ■ y q ) 

t=i 

for k > 1, do is an augmentation map extracting the constant term, and <9_i = 0. Then 
it is easy to verify that d k _i ° d k = 0. 

Proposition 3.1 TTie c/iam complex M = (M k , d k ) is a free resolution of the trivial 
1- dimensional left U -module C. 

Proof. Case 1. Suppose first that £ is abelian. Then, by the Poincare-Birkhoff-Witt 
Theorem, we have 

(3.3) [/(£) ~ S(£ ) ® A(£i) 

as a C-vector space, and the differential maps are simplified to 

<9 fc (w <S> (x! A ■ ■ ■ A x p ) <g) (y 1 ■ ■ ■ y q ) 

v 

= J^(-l) s+1 (w • x s ) <g> (xi A • • • A x s A ■ ■ ■ A x p ) <g> (y : • • • y q ) 

s=l 

q 

+ (-l) p ^(w-y t ) <g> (xi A--- Ax p )® (yi- ■ -yf-yq)- 



t=i 

We define the homotopy map D : M k — > M fe+1 by 

• • -tt s vi • • -f t ) <g) (xi A • • • Ax p ) <g) (yi ■ ••yq)) 

s 

= ■ ■ -Ui- ■■ u s vi - ■ ■ v t ) ® (ui A xi A • • • A x p ) <g) (y ± ■ ■ - y q ) 

i=i 
t 

= ^(-1) p+ *^(mi • --u s vi ■ ■ -Vj ■ ■ -v t ) ® (xi A • • • A Xp) ® (Dj-yi • • 
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where it, G £o an d v j £ £i- Then it is easy to verify that 

Dd + dD = (p + q + s + t)l M , 

which implies that our complex M = (M^, dk) is exact in this case. 

Case 2. Let £ be any Lie superalgebra. Recall the construction of the universal 
enveloping algebra of £. Put U n (£) = tt(0" =o £® fc ), where tt : T(£) -> [/(£) is the 
canonical projection map and X^f 1 = U p ~ n (£) <8> C n (£) for < n < p. 

Consider the complex 

(3.4) — Xf> ^ ■ ■ ■ X W -i+ C — > 0. 

We will use the induction on p to prove the exactness of (|3.4j ), which would imply the 
exactness of our original sequence. 

Note that the exactness of (|3.4j ) is equivalent to that of the following sequence: 

(3.5) — > X® ■ . . - X[ p) X^ /C — > 0. 

For p = 0,1, our assertion is clear. Suppose (|3.5| ) is exact for p — 1. Consider the 
diagram 



0^ o x?^ ^ x^/c ^0 

kj. J, J, 

^ ■■■ ^ x{ p) ^ x( p) /c ^o 

o^ xjr } ^ ■■■ $ x^/x^ 4. x®/x!r 1) -o. 

J, J, J, 



Note that 

XW/Xj?- 1 ') ~ (E7 p - n ( ( C)/C/ p - n - 1 ( ( C)) ® C n (£) (0 < n < p - 1) 
as a C- vector space, and that, by the Poincare-Birkhoff-Witt Theorem, 
C/p-"^)/^-"- 1 ^) ~ S fc (£ ) <g> A'(£i) 

fc+i=p— n 

by the canonical homomorphism and X^ ~ C ® C p (£). Finally, the differentials in 
bottom row are given by 

^(u ® (xi A • • • A x p ) ® (yi • • • y,) + X^) 



(3.6) 



= ^(-l) s+1 (w • x s ) <g> (x x A • • • A z, A • • • A x p ) <g> (^ • • • y q ) 

s=l 

q 

+ (-l) p Y,( u ■ Vt) ® O&i A • • • A x p ) ® • • • y t ■ ■ ■ y q ) + X 



(p-i) 

n-l 

t=l 
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Thus the bottom row can be viewed as a subcomplex of the complex considered in 
Case 1, and hence it is exact. Therefore, the middle row is exact, which completes our 
induction argument. 

Since C(£) is a C- vector space, U <S> C(£) is the direct sum of free [/-modules. 
Hence, U <8> C(£) is a free resolution of the 1-dimensional trivial left [/-module C. □ 

Let V be a right [/-module. The homology module H k (£, V) is defined to be 
(3.7) H k (£,V) = Tor^(\/,C) = H k {V ® v M), 

where M = (M k , d k ) is the free resolution of the 1-dimensional trivial left [/-module C 
given by Proposition [JO. Since we have 



V® V M^V ®uU® c C k {£) = V® c C k {£) (k > 0), 

and 

V ® v C S V, 

as C- vector spaces, we obtain the following standard complex for the homology of Lie 
superalgebras: 



(3. 



V ® c C fc 0C) V ® c 0-i(£) • • • ^ V ® c C (£) = V — >• 0, 



where the differentials d k = ly ®u 9 k : V" ®c Cfc(£) — ► V" ®c Cfc-i(£) are given by 
(3.9) 

d k (v ® (a;i A • • • A x p ) <g> (yi ■ ■ ■ y q )) 

= (- 1 ) S+tv ®([x s ,x t }Ax 1 A---Ax s A---Ax t A---Ax p )®(y 1 ---y q ) 

l<s<t<p 
P Q 

+ Y X^ -1 )^ ® fci A ■ ■ ' A ^ A ■ ' " A x p) ® (t x *> vt]vi ■■■Vf-yq) 

s=l t=l 

- v ®([y^yt]^x 1 A---Ax p )(g)(y 1 ---y s ---y t ---y g ) 

l<s<t<q 
P 

+ ^2(-l) s+1 (v ■ x s ) ® (xx A • • • A x s A • • • A Xp) <g> (y x ■ ■ -y q ) 

8=1 



+ (-l) p ■ ?/ t ) <g> (xi A ■ ■ ■ A x p ) ® (yi ■ • • £ • ■ ■ y q ) 



t=i 



for u G V, Xj G £q, yj G £i. In particular, when V = C is the trivial 1-dimensional 
right [/-module, the standard complex ( |3.8| ) is reduced to the complex fl2.8| ) given in 
Section 2. 



As a corollary of Proposition 3.1, we obtain: 
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Corollary 3.2 Let A be a countable abelian group with a coloring map 6 and V = 
®aeA^a be a 6 -colored superspace. Let £ be the free Lie superalgebra generated by V . 
Then we have 



(3.10) H k (2) 



V ifk = l, 
ifk>2. 



Proof. Note that the universal enveloping algebra of £ is isomorphic to the tensor alge- 
bra T(V) generated by V. Consider the following free resolution of the 1- dimensional 
trivial left T (V>module C: 

-> T{V) <g> V ^ T(V) T(V)/T(V) <g> V -> 0, 

where ip is the natural inclusion map(=multiplication map) and tp is the augmentation 
map. 

By tensoring with trivial right T(l / )-module C, we get 

-> C ® T (v) T(V) ®V l ^C ® T {v) T(V) ^ C <g> C -> 0. 

Since 1 <g) </? is a zero map, Hi(C) = Ker (1 <g> </?) = C <8>r(v) T(V) <8> V — V, and 
if fc (£) = for k > 2. □ 



4 Free Lie Algebras and Free Lie Superalgebras 

Let A be a countable abelian group with a coloring map 6 and T be a countable 
abelian semigroup satisfying the fmiteness condition given in Section 2. Let V = 
©(«a)erx^^(a,«) be a (r x ^-graded ^-colored superspace with finite dimensional 
homogeneous subspaces, and let £ = ®, a ) Grx _4 £(a,a) be the free Lie superalgebra 
generated by V. Suppose that a group G acts on V preserving the (r x v 4.)-gradation. 
Then the action of G on V can be extended to £ by Lie superalgebra automorphisms. 

As we have seen in Section 3, we have i?i(£) = V and -£/&(£) = for k > 2. Hence, 
for any g G G, the generalized denominator identity for the free Lie superalgebra £ is 
equal to 

J] exp (-f2l^9 k \£(a,a))E k ^ a A 

^4 1) (a,a)eFxA V k=l / 

= l-sch s \/=l- ]T str(^|V( a , a) )^. 

(a,a)eTxA 

For any g e G, let P(V,r x .A) = {(a, a) e T x A \ str(#|V (Q;a) ) ^ 0} and let 
{(ai,Qj) | i,j = 1,2,3, •••} be an enumeration of the set P(V,T x A). For each 
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(r, b) G T x A, we denote by T(r, b) the set of all paritions of (r, 6) into a sum of 
(ccj, aj)'s as defined in ( |2.15| ), and let W g (r, b) be the Witt partition function as defined 



in ( |2.16| ). Then our supertrace formula ( |2.17| ) yields 



(4.2) str(^|£ (a , a) ) = Y. - d ^ {d) S ^IT^II^I^))^- 

d>0 s&T(r,b) ° i,j 

(a,a)=d(r,b) 

Remark. The supertrace formula ( |4.2|) is a generalization of the superdimension for- 



mula for the free Lie superalgebras given in p9| and pO . 



Example 4.1 In this example, we will consider the simplest application of our super- 
trace formula to free Lie superalgebras. Consider the superspace V = V © V\ with 
dim Vq = r and dim V\ = s for some r, s G Z >0 , and let £ be the free Lie superal- 
gebra generated by V . By setting degf = 1 for all v G V, the free Lie superalgebra 
£ has a Z >0 -gradation £ = 0^° =1 £„ induced by V. Let G = GL(V ) x GL(Vi) = 
GL(r) x GL(s). For (g, h) G G, we denote by t g = tr(g|Vo) and th = tv(h\Vi). Since 
P(V,Z >0 ) = {1} and str({g,h)\V) = tr{g\Vo) - tr(/i|Vi) = t g -t h for all /i) G G, the 
Witt partition function (^) is given by 

— IV 1 

w M (n) = { -^r Jl (t 9 - t h ) n = -(t 9 - t h ) n . 

Hence, by the supertrace formula (|4.2|) , we have 

(4.3) str((g,h)\& n ) = -^2n(d)(t g -t h )^. □ 

d\n 

Next, we will discuss the relation of the free Lie algebra and the free Lie superalgebra 
generated by the same vector space V = ®( a a ) g r X A on wm ch a group G acts 
preserving the (r x ^4)-gradation. By neglecting the coloring map 9 on A, consider V 
as a (r x ^4)-graded vector space, and let L = ®( a ,a)erxAL( a ,a) be the free Lie algebra 
generated by V. Then the action of G on V can be extended to L by Lie algebra 
automorphisms. Hence, for any g G G, the generalized ednominator identity for L is 
equal to 

II ex P [-£itr(/|L (aja) )e^) 

(4 4) (a,a)erx.4 V fe=l / 

= l-ch fl V = l- Yl tr W«,a))e (a ' a) . 

Since = ?/>(a)e( Q ' a ) and sbr{g\V M ) = if>(a)tr(g\V M ) for all {a, a) e T x A, 

the right-hand sides of (|4.1|) and (|4.4|) are the same. Moreover, note that, for aiinxn 
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complex matrix A with eigenvalues x\, ■ ■ • , x n , Lemma 2.1 implies 
ex P (Ej^*)**) ex P (E ^(-l) fe tr(A fc )^ 

~ n i _ x .t n i + x . t ~ n ! _ ^ 

oo / oo 1 \ 

= £ K(xl, a*)*" = exp £ ^tr(A 2fc )^ . 

n=0 \/b=l / 

Therefore, the left-hand side of Q4.4| ) is equal to 

J] exp ( - f; ~tr(^|L (Qj0) ) e ^> a ) J J] exp ( - £ ^tr(g k \L m )e k ^ 

(a,a)£TxAo V fc=l / (f3,b)^rxAi V fc=l 

n ex p(-E^ fe i^))^ (a ' a) ) 

{a,a)eTxAo V fc=l / 

x J] ex P (-£^tr(/|L (A6) )^^)) 

{/3,b)eTxAi V fc=l / 
II «xp(-f;itr(/|L (Qia) )E fc ^)) 

(a,a)erxA) V fc=l / 

II exp^itr^lL^))^) 
)erx^j \fc=i / 

n ex pf-E^(^i^))^ fe(A6) l 

lerxA V k=i ) 



X 

(/3,b)erx^i 



= II ex P ( - E ^(a)trO/*|L (aia) )^M 
(«,«)erx^ V k=i / 

x II e^(-f;itr(^|L W) )^V 

(a,o)=2(/3,6) V fc=l / 

(/3,b)erx^i 

Hence, for any g e G, the generalized denominator identity for the free Lie super- 
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algebra £ is the same as 



(4.5) 



J] exp (-f2lstx{g k \£ {a , a) )E k ^A 

{a,a)eFxA V fe=l / 

= 1- Yl M9\V(a,a))E^ a) 
(a,a)eFxA 

= II ^P(-E^(«)tr(/I^( a ,a))^ (a ' a) ) 
(«,a)erx^i V k=l J 

II exp(-f Itr(^|L (Ab) )^-)), 

a)=2ffl,6] V fc=l / 



(a,a)=2(f3,b) 
((3,b)eFxAi 



which yields: 



Proposition 4.2 Let A be a countable abelian group with a coloring map 9 and V be a 
countable abelian semigroup satisfying the finiteness condition given in Section 2. For a 
(r x A)-graded superspace V = ®/ a a ) grxy!l V(a,a) with finite dimensional homoge 



neous 



subspaces, let L = ®( a a ) erx ^ £(a a) be the free Lie algebra generated by V, and let 
a)eTxA £(<x,a) be the free Lie superalgebra generated by V. Suppose that a 
group G acts on V preserving the (T x A) -gradation. Then, for any g G G, we have 



(4.6) str(g\£ (a , a) ) = ^(a)tr(g\L (a , a) ) + 



09,6) y 



(a,a)=2(/3,b) 
(f3,b)£TxAi 



□ 



Remark. When g — 1, we recover the superdimension formula 
(4.7) sdim£ (aia) = ip(a) dimL (Qja) + ^ dimL (( g )6) , 

(a,a)=2(J3,b) 
(f3,b)eFxAi 



which was obtained in EO . 



5 Decomposition of Free Lie Superalgebras 

In this section, we investigate the structure of the free Lie superalgebra generated by 
the natural representation of the general linear Lie superalgebra gl(k, I). For k, I G Z> , 
let £ = gl(k, I) be the space of all (k + I) X (k + I) matrices, and set 



(5.1) 



A 
D 



A is a k x k matrix and D is an I x / matrix 
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(5.2) £x 



B 
C 



B is a k x / matrix and C is an / x matrix 



Then £ = £ © £i and £ becomes a Z 2 -graded Lie superalgebra called the general 
linear Lie superalgebra with the superbracket defined by 



(5.3) 



[X, Y] = XY — {-l) af3 YX for Ig£„, Ye £ /3 , a,/3e Z 2 . 



A Z 2 -graded superspace V = V" © V\ is an Z-module if there is a Z 2 -graded Lie 
superalgebra homomorphism 



(5.4) 



: £ — ► gl(V) = gl(k',l'), where k! = dimV , I' = dimVi. 



For £-modules V^ 1 , • • • , K™, the tensor product V 1 ® 
with the action of £ defined by 

n 

x ■ (v ± <g> • • • <g> u n ) = ^(-l)***^**^ 



V™ becomes an £-module 



x • f j <g> • • • ® v n 



for ar G £, v { E V\ 

We recall some of the basic theory of symmetric functions. A partition is any 
(finite or infinite) sequence A = (Ai,A 2 ,---) of non-negative integers in decreasing 
orders Ai > A 2 > • • ■ and containing only finitely many non-zero terms. The non-zero 
Aj's are called the parts of A. The number of parts is the length of A, denoted by /(A), 
and the sum of parts is the weight of A, denoted by |A|. If |A| = n, then we say that 
A is a partition of n, denote by A h n. We also use the notation which indicates the 
number of times each integer occurs as a part. That is, A = (l mi , 2 m2 , • • • ) means that 
i appears as a part of A exactly rrii times. Each partition A uniquely determines a 
Young diagram. The conjugate of a partition A is the partition A' whose diagram is the 
transpose of that of A (i.e., the diagram obtained by reflection in the main diagonal). 
There is a partial ordering on the set of parititions of n, called the natural (partial) 
ordering or the dominance relation, which is defined as follows: 

k k 

A > ji if and only if \ > ^ for all k > 1 . 



i=i 



i=l 



Let Xi, • • • , Xfc be the indeterminates and A a partition of n with /(A) < k. A Schur 
function corresponding to A is a symmetric function in x±, • • • ,Xk defined by 



(5.5) 



S x ( x ) = S x (xi ■■■ ,x k ) = 



X 



Xj+n-j 



\X 



n-j 



For a partition /i = (fj,i, /i 2 , • • • ), we define 

(5.6) p^x) =p fl (x ir -- ,x k ) = JJp w (xi,-- - ,x k ), 
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where p w (xi, • • ■ , x^) = x j' is the /^rth power sum. It is known that the set of Schur 
functions { S\(x) | /(A) < k} (resp. {p\(x)\ /(A) < k}) forms a Z-basis (resp. Q-basis) 
of the ring of symmetric functions in x\, ■ ■ • , x^, and { S\(x) | /(A) < k, A h n } (resp. 
{ Pa 0*0 1 K^) — A h n}) forms a Z-basis (resp. Q-basis) of the subspace of symmetric 
functions in x±, ■ ■ • ,Xk with homogeneous degree n. The functions S\(x) and p\(x) are 
related as follows: 



Lemma 5.1 ([[!§]) For each partition p of n, we have 
(5-7) p p (x) = ^2x P xS x (x), 



Ahn 



where x P \ is the character value of the irreducible representation of S n corresponding to 
A at the conjugacy class of type p. □ 



A partition A is called a (k,l)-hook partition if A&+i < /. Set H(k,l;n) = { A h 
n I Afe + i < / }. For convenience, we will often write H for H(k, I; n). For A G H(k, I; n), 
a (k, I) -hook Schur function in Xx, • • • , x^, yi, ■ • • ,yi corresponding to A is a function 
defined by 

(5.8) HS x (x,y) =HS x (x ir -- ,x k ,y ir -- ,y t ) =^2S lx (x)S x >/n>(y), 

where Sy/^'(y) is the skew Schur function corresponding to A',/i' (cf. M, [EB], [p2J). 
Equivalent ly, 

(5.9) HS x (x,y) = ]>>„(*) ( ^N^J5 v (y) J , 

where N^, v is the Littlewood- Richards on coefficient corresponding to A', fjf, v. The com- 
binatorial interpretation of the Littlewood-Richardson coefficients is given as follows: 
for given partitions \ \l,v satisfying |A| = + \v\ and \ > fa for all i, is the 
number of ways the Young diagram for fi can be expanded to the Young diagram for A 
by a strict v -expansion (if the above conditions do not hold, = 0). For a partition 
v = {yi, • • • , z/fc), a v-expansion of a Young diagram for \l is obtained by first adding v\ 
boxes to the Young diagram for fi with no two boxes in the same column and putting 
the integer 1 in each of these V\ boxes; then adding similarly zaj boxes with a 2, contin- 
uing until finally v k boxes are added with the integer k. The expansion is called strict 
if, when the integers in the boxes are listed from right to left, starting with the top row 
and working down, and one looks at the first t entries in this list (for any t between 1 
and \v\), each integer p between 1 and k — 1 occurs at least as many times as the next 
integer p + 1. 

In ||, Berele and Regev showed that, for a Z 2 -graded superspace V = Vq © V\ 
viewed as a gl(k, Z)-module (k = dimVo, / = diml^), V® n is completely reducible 
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as a gl(k, /)-module and its irreducible components are parametrized by (k, /)-hook 
partitions A, denoted by V\. 

Let -C(V) be the free Lie superalgebra generated by V. Set £(V)( n)0 ) = £(V) D 
(V®") , £(y) ( „,i) = £(F)n(V®»)i and £ n (V) = £0/) (n , 0) ©£(y) (nil) . Thus £(F) has 
an (N x Z 2 )-gradation: £(V") = 0mxz 2 ^(^)m- The homogeneous subspace £ n (V) 
is a Z)-submodule of V® 11 , where the action of x G gl(k, I) induced from V® n is 
given by 

X ■ [Vi, [v 2 , [• ■ ■ [Vn-l,V n ] ■■■]]] 
n 

= J2(-l) de ^« de ^[ Vl , [■ ■ ■ [xvi[. ■ ■ [v n . u v n ] ■ ■■}))}, 
1=1 

and it is completely reducible with its irreducible components parametrized by X's G H. 

^From now on, we simply write £ n , £( n ,o) an d £(n,i) f° r £n00> £(V)(n,o) an d 
£(V)( n) i), respectively. For each A G H(k,l;n), let c\ be the multiplicity of V\ in 
£ n . To compute c\, we would like to apply our supertrace formula. By exponentiating 
the action of the Lie algebra gl(k,l) = gl(k) x gl{l) on V, the group GL{k) x GL(l) 
acts on V, and hence V® n and £ n (V) become GL{k) x GL(/)-modules. We first recall: 

Proposition 5.2 ([|]) Let A G H(k,l;n) and V\ be the corresponding irreducible 
representation of gl(k,l). Suppose that g = (go,gi) G GL{k) x GL{1) has eigenvalues 
Xi, • • • ,Xk for g and yi, - • ■ ,yi for g\. Then we have 

(5.10) tr{g\V x ) = HS x (x,y). 

□ 

Our supertrace formula (|4.2|) for free Lie superalgebras yields 
(5.11) 

str(^|£ (n , a) ) = \^ d ) E ~ 1)! str(^|^ ) s °str(^|Vl) s S 

d\{n,a) seT((n,a)/d) 

where 

T(m, 0) = { (s , si) | s + s\ = m, s± = (mod 2) }, 

T(m, 1) = { (s , si) | s + si = m, s\ = l (mod 2) }. 
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Hence, for g = (go, gi) G GL(k) x GL(l), we have 
tr(#| £(„,<))) = 8tr(^|£( n> o)) 



d|(n,0) s£T((n,0)/d) 

= E>> E 



(W-i)i 



rf|(n,0) s6T((n,0)/d) 



(-l) ai p d (aO> d (s/) 



si 



= E>> E ^(-ir(E^w)(E 

d|(n,0) s£T((n,0)/d) \Ahds„ / \/ihdsi 



E^) E M ^(-D' 1 E X\'° ) xf 1) Sx(x)S li (y) 



\ 



d|(n,0) seT((n,0)/d) 



Ahdso 
\/ihdsi 



/ 



and 



tr(</ | £(„,!)) = -str(^|£ (n> i)) 

= - E ^(<0 E ^j^str(^|Vo)'°8tr(^|V 1 )« 

d|(n,l) seT((n,l)/d) 

= " E E ^r^(- 1 ) Sl tr(^lK,) Sl, tr(^|\/ 1 ) 

d|n seT((ra,l)/d) 
d:odd 

E \^ d ) E ^r^(^>^) si 

| a |-l)! 



d\n s€T((n,l)/d) 
d.odd 



= E>) E 



d ' s 1 

d|n s€T((n,l)/d) 
d.odd 



E xT»s»W E *r%w 



\Xhdso 



\fihdsi 



E> E ^ EWww 



d|n s€T((ra,l)/d) " \ Ahds 

d:odd Vhdsi 
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It follows that 
(5.12) 



tr(#|£ n ) = tr(^|£ M) ) + tr(g\£ M ) 

.. [ \ 



d ^— ' s 

d\n so+si=n/d 
d.odd 



X\-ds Q I 
\fihds! / 

E ~Ad) £ (-l)-M^M E Xx S ° ) xf 1) Sx(x)S tl (y) 



d , s 

d\n so+si=n/d 
d:even 



\ Xhds I 
\fj}-dsi / 



^2 a\^S x (x)S fl (y), 



l(X)<k 
l(p)<l 
|A| + H=n 



where 



^ //(d) (\s\ - 1)\ {d s 0) (dSl) 



d\n so+si=n/d 
d.odd dso = \X\ 



d|n so+si=n/d 
i:even dso=| A| 

I V u ( d ) ( ^(d-DM (dl^l/^ (rf lMl/ d) 



^ J (\M/d)K\,\/d)\ 

Note that cl\^ is the multiplicity of W\ ® W 7 ^ in £ n , where Wa (resp. H 7 ^) is the 
irreducible GL(fc)-module (resp. GL(Z)-module) corresponding to A (resp. /jl). Recall 
that c\ is the multiplicity of V\ in £ n . By Proposition |5T2], we have 



(5.13) tr(g\£ n ) = J2cxHS x (x,y). 

Therefore, we obtain: 



Proposition 5.3 If X = (t\, T2, ■ ■ ■ , t>, Tk+i, ■ • ■), then we have 
(5-14) c x = a XoXl - c ^ N \' \v 

A*o>Ao 

where A = (n, ■ ■ ■ , r k ) and Ai = (r fc+1 , •••)'. 
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Proof. Comparing the equations fl5.13| ) and ( |5.14j) , we have 
^2 CpHSp(x, y) = c x HS x (x, y) + ^ c n HS n( x , v) 



c\ I ^2 s A x ) s x/»'(y) J + E c n HS n( x ' y) 

\v<\ / 



I^X 

c A (s Xo (x)s Xl ( y ) + (*)) + E c m ( E s T (x)J2Kis„(y) 

( \ 



+ E c ^ 



cx 



\ MO>Ao 



S Xo (x)S Xl (y) + (*)' 



J 

= E a ^ s *( x ) s »(y)- 

l(X)<k 

i(n)<i 
|A[+|mI=" 

Since { S x (x)S IJj (y) \ 1(A) < k, Z(/i) < I } is linearly independent ([§), we obtain 

a XoXl =c x + c ^ N x' x^ 

H&H-{X} 

fJ.Q>X Q 

which completes the proof. 

If /(A) < k (or |A | = n) for A 6 H(k, I; n), then (£T§ yields 

(5.15) c x = QxqXi = - ^2 Kd)x { f /d) ■ 



□ 



d\n 



If all ca's are known for m < |Ao| < n, then for any A G i?(A;, Z; n) with |Ao| = m — 1, 
we have 



ca — OA Ai _ 



E ^ 



(5.16) 



Ate/MA} 

A*o>Ao 



a A A! - E C » N X' Xxi 



MO>Ao 
|/io|>m 



since /io > Ao and |/io| = m — 1 imply /x = Ao, and in this case, Ny x = unless A = ll 
(cf. [@). Following the above inductive step, we can completely determine the value 
of all Ca for A e H(k, I; n): 

Proposition 5.4 Under the above hypothesis, we have 
(5.17) £ n (V)= Vf c \ 

Aeff(fc,Z;«) 
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where V\ is the irreducible gl(k, I) -module corresponding to A and c\ is determined by 
(EH- □ 



Remark. Note that the multiplicity c\ is given explicitly in closed form when /(A) < k 
and is given by a recursive formula otherwise. 

Corollary 5.5 (cf. ||, |2~3[| ) Under the above hypothesis, ijV\ = 0, then we have 

(5.18) £ n (V) = Vr\ 

Xhn 
l(X)<k 

(5.19) CA = ±x>(d)x<r /-) , 

where V\ is the irreducible gl(k)-module corresponding to X. □ 



6 Generalized Kac-Moody Superalgebras 

The generalized Kac-Moody superalgebras arise naturally in the context of Monstrous 



Moonshine (B, B), automorphic forms with infinite product expansions (M, [IS 



p0||), and the string theory ( ||22|| ). In this section, we give a conjectural formula for the 
homology modules over (the negative part of) generalized Kac-Moody superalgebras, 
and derive a closed form supertrace formula for generalized Kac-Moody superalgebras 
with group actions. 

Let J be a countable (possibly infinite) index set. A real square matrix A = (fly)ijgj 
is called a Borcherds-Cartan matrix if it satisfies: (i) an = 2 or an < for all % G J, 
(ii) Ojij < if i j, and a^ 6 Z if a„ = 2, (iii) = implies = 0. We say that an 
index % is real if an = 2 and imaginary if ajj < 0. We denote by I re = {i G I\ an = 2}, 
pm _ |^ ^ j | a .. ^ Let m = (m 8 G Z >0 | z G J) be a sequence of positive integers 
such that m, = 1 for all i G J re . We call m the charge of A. In this paper, we assume 
that the Borcherds-Cartan matrix A is symmetrizable, i.e., there is a diagonal matrix 
D = diag(si\ i G I) with Sj > (i G I) such that DA is symmetric. 

Let C = (0ij)i,jei be a complex matrix satisfying OijOji = 1 for all i,j G /. Thus 
we have da — ±1 for all z G J. We call i G / an even index if 0jj = 1 and an odd index 
if On = — 1. We denote by / e,;en (resp. J odd ) the set of all even (resp. odd) indices. 
We say that a Borcherds-Cartan matrix A = (ay)j je / is colored by C if an = 2 and 
9a = — 1 imply ajj are even integers for all j G /. In this case, the matrix C is called a 
coloring matrix of A 



26 



Let f) = (® ie/ Chi) © Cdi) be a complex vector space with a basis {hi, di\ i G 

J}, and for each i G / define a linear functional on G f)* by 

(6.1) a i(hj) = ciji, a,i{dj) = 5ij for all j G J. 

The free abelian group Q = @ ie/ Z«j generated by a^s {i G /) is called the root lattice 
associated with A. Let II = {a>i\ i G /} and B be a basis of f)* extending II. Set 
B' — B \ II. Since A is assumed to be symmetrizable, there is a symmetric bilinear 
form ( | ) on fj* defined by 

(ai\aj) = Sidy for i, j G /, 

(6.2) (\\ ai ) = X( Si hi) for AGS', 

(A|/i) =0 for A,/i G 

(cf. see, for example, |24j]). Let Q + = Siei" ^>o a i an d Q~ = —Q + - There is a partial 
ordering > on Q given by A > fi if and only if A — n G Q + - The coloring matrix 
C = (0ij)i,jei defines a bimultiplicative map 9 : Q x Q — > C x on Q by 

#(«,, a,) = 6^- for all i, j G /, 

(6.3) 0(a + 0, 7 ) = 0(a, 7W,7), 

0(a,/3 + 7) = 0(a,/3)0(a, 7 ) 

for all 7 G Note that, since 0f/0jj = 1 for all i,j G /, satisfies 

(6.4) 0(a, /3)9(J3, a) = 1 for all a,/3eQ. 

That is, 6* is a coloring map on Q. In particular, 6 (a, a) = ±1 for all a G Q. We say 
a G Q is even if ^(a) = 0(a, a) = 1 and odd if t/>(ck) = 0(a, a) = — 1. 

The generalized Kac-Moody superalgebra q = g(A,m,C) associated with a sym- 
metrizable Borcherds-Cartan matrix A = (ar,)?, ): ,£/ of charge m = (mj| i G /) with 
a coloring matrix C = (9ij)ij £ i is the ^-colored Lie superalgebra generated by the 
elements hi, di {i G /), e^, (i 6 J, k = 1, 2, • • • , mj) with the defining relations: 

[hi, hj] = [hi, dj] = [di, dj] = 0, 

[hi, Cjl\ CLijCji, [hi, fjl] dijfjh 

^ ^ [di, Cji] = $ij e jh [di, fji] = Sijfjh 

[&ik, fjl] = SijSklhi, 

(adeikf-^ (eji) = {adfik) 1 '" 13 (fji) = if a u = 2 and i + j, 

[eik, eji] = [fik, fji] = if Oij = 

for i, j G I, k — 1, ■ • • , mj, / = 1, • ■ ■ , m.,-. 

The abelian subalgebra 1) = (® ie/ Chi) @ (©.j e / Ccij) is called the Cartan subalge- 
bra of g, and the linear functionals 6 I)* (i 6 /) defined by ( p.l\ j are called the simple 
roots of g. For each i G I re , let G GL(f)*) be the simple reflection on fj* defined by 

7"i(A) = A — \(hi)oti for A G f)*. 
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The subgroup W of GL{\)*) generated by the r^s (i G I re ) is called the Weyl group of 
the generalized Kac-Moody superalgebra g. 

The generalized Kac-Moody superalgebra g = g(A, m, C) has the root space decom- 
position g = agQ g a , where 

Q a = \x G g| [h,x] = a(h)x for all h G f)}. 

Note that g Qi = Ce^i©- ■■@Ce i>m and g_ ai = Cf itl ®-- -®Cf i)mi . We say that a e Q x 
is a rooi of g if g Q 7^ 0. The subspace g a is called the root space of g attached to a. A 
root a is called rea/ if (a\a) > and imaginary if (a|a) < 0. In particular, a simple 
root «j is real if an = 2 (i.e., % G J re ) and imaginary if an < (i.e., % G J im ). Note that 
the imaginary simple roots may have multiplicity > 1. A root a > (resp. a < 0) is 
called positive (resp. negative). One can show that all the roots are either positive or 
negative. We denote by $, $ + , and $~ the set of all roots, positive roots, and negative 
roots, respectively. We also denote by $ (resp. $1) the set of all even (resp. odd) 
roots of g. Hence, for example, $q will denote the set of all positive even roots of g. 
Define the subalgebras g 1 * 1 = © a6$ ± Qa- Then we have the triangular decomposition: 

g = g- © t) ©g + . 
We can define a nondegenerate symmetric bilinear form on f) by 
(6.6) (hi\h) = — 014(h) and (di\dj) = 

for all h G t), i,j G / (cf. |3H|). The symmetric bilinear form ( | ) on f) defined by (|6.6|) 
can be extended to a nondegenerate, supersymmetric, and invariant bilinear form on 
the generalized Kac-Moody superalgebra g: 

Proposition 6.1 (cf. p0[ ) Lei g = g(A,m, C) fre i/ie generalized Kac-Moody 

superalgebra associated with a Borcherds-Cartan data (A,m,C). Then there exists a 
nondegenerate bilinear form ( | ) on g satisfying the following conditions : 

(a) T/ie bilinear form ( \ ) is supersymmetric, i.e., we have 

(x\y) = 9(f3,a)(y\x) for x G g a ,y G g p . 

(b) T/ie bilinear form ( \ ) is invariant, .i.e., we have 

([x,y]\z) = (x\[y,z]) forx,y,zeg. 

(c) The bilinear form ( | ) | (, is given by ( |6.6| ) . 

(d) (Qa\ep) =0 z/« + /3^0. 

(e) T/ie roo£ spaces g a and g_ a are nondegenerately paired with respect to ( | ), ie., 
£/ie bilinear form ( | )| 0a + _ a zs nondegenerate. 

(i) If <p : Q t) is a linear map satisfying 0(a*) = Sj/ij /or a// 2 G J ; £/ien we /jave 
= 6 l (a,a)(x|?/)0(a) /or x G g a , y G g_ Q . □ 
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A g-module V is called i)-diagonalizable if it admits a weight space decomposition 
V = ©^eir V v where 

Vf, = {v G V\ h ■ v = fi(h)v for all h G h}. 

If Vfj, 7^ 0, then \i is called a weight of V and is called the p-weight space. We denote 
by P(V) the set of all weights of V. When all the weight spaces are finite dimensional, 
we define the character of V to be 

(6.7) chV = ( dim ^) efl > 

where e M are the basis elements of the group algebra C[fj*] with the multiplication 
given by e^e" = e^ for p, v G f)*. 

We denote by O the category of f)-diagonalizable g-modules with finite dimensional 
weight spaces such that there exist a finite number of linear functionals Ai, • • • , A s 
satisfying P(V) C U| =1 D(Aj), where D(\) = {p G f)*| /J < A}. The morphisms in O 
are the usual g-module homomorphisms. 

The most important example of the g-modules in category O may be the class of 
highest weight modules. An f)-diagonalizable g-module V is called a highest weight 
module with highest weight A G f}* if there is a nonzero vector v\ (z V such that (i) 
£ik ■ V\ — for all i G /, k — 1, • • • , m^, (ii) h ■ v\ — \{h)v\ for all /i G f), (iii) 
= £7(g) • t>A- The vector v\ is called a highest weight vector. For a highest weight 
module V with highest weight A, we have (i) V = U(g~) ■ v\, (ii) V = © M<A V^, 
V\ = Cv\, and (iii) dimV^ < oo for all p < A. 

Let b + = rj © g + , and let C\ be the 1-dimensional b + -module defined by h ■ 1 = 
A(/i)l for all h G t) and g+ • 1 = 0. The induced module M(A) = U(q) <S>u(b+) C A is 
called the Verma module over g with highest weight A. Every highest weight g-module 
with highest weight A is a homomorphic image of M(A) and the Verma module M(A) 
contains a unique maximal submodule J(A). Hence the quotient V(X) = M(X)/J(X) 
is irreducible. 

Let V be a g-module in category O. We define the Casimir operator on as follows 
(cf. p4jl , [j30|1 ). Take a linear functional p G f)* satisfying p(/i,) = for all i & I. Such 
a linear functional is called a VKe?// vector of g. Note that (p|ctj) = p(sihi) = i(a,|aj) 
(i G /). For each positive root a G $ + , choose a basis {e^l i — 1, ■ - • , dimg a } of the 
root space g a and let {e_ a \ i = 1, ■ ■ • ,dimg Q } be the dual basis of g_ a in the sense 
that (e£ \e_ a ) = 5ij6(a,a). Then for a weight vector u G V\ of weight A, the Casimir 
operator Q on is defined by 

dimg a 

(6.8) n(«) = (A + 2p|AH + 2^^ e« e g>« A - 

The Casimir operator commutes with the action of g on V. Recall that a weight vector 
f G V is called a primitive vector if there is a g-submodule W of V such that v W 
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and ■ v G W for alH G I, k = 1, 2, • • • , (cf. [[H]]). The Casimir operator satisfies 
the following properties: 

Proposition 6.2 (a) If V is a highest weight g-module in Category O with highest 
weight A, then we have 



(6.9) Q = (A + 2p\A)L 



v ■ 



re 

re i~i Todd 



(b) 7/t> is a primitive vector with weight X, then there exists a Q-submodule W C V 
such that v ^ W and 

(6.10) Q(v) = (X + 2p\X)v modW. □ 

Let P + be the set of all linear functionals A G h* satisfying 

{X(hi) G Z> for all i G J r 
A(/ii) G 2Z> for all i E I re D I 
X(hi) > for all i G P m . 

The elements of P + are called the dominant integral weights. For a dominant integral 
weight A G P + , let 

$ + (A) = {0 = k i®i G Q + \ ( A N) = for h > 1, 
(6.12) ig/™ 

(aj|aj) = for fc,, fcj > 1, i ^ j, («j|aj) = for ki > 2}. 
For such an element (3 G <£> + (A), we denote \(3\ = ^2 ieI i m ki and 

(6,3) e(W = n (?) n C^ -1 )- 

Then the character of the irreducible highest weight module ^(A) with highest weight 
A G P + is determined by the Weyl-Kac-Borcherds formula : 

Proposition 6.3 (@, [fTTH ) 
(6.14) 



/36*+(A) 

Letting A = 0, we obtain the denominator identity: 



( 6 .15) ) = y (_i)«(«)+l/»l e (^ c «(^/s)-P □ 

/3e<i> + (0) 
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Corollary 6.4 A highest weight g-module V with highest weight A G P + and highest 
weight vector v\ is irreducible if and only if it satisfies: 

,A(7u)+l n -f ■ rz jre 

(6.16) hk " A "° lhGi ' 

fik ■ v A = (k = 1, • • • , mt) if A{hi) = 0. □ 

Suppose a group G acts on the generalized Kac- Moody superalgebra q = g(A, m, C) 
by Lie superalgebra automorphisms of 0± preserving the root space decomposition. 
Assume further that str(g|g a ) = str(g|g_ Q ) for all g G G and a G Q + . We would like 
to apply our supertrace formula ([2.171) to derive a closed form formula for str(g|g Q ) 
(geG,ae Q). 

Let J be a finite subset of 7 re , and let $j = $ n (X^ e j $j = $ ± fl $,/, 

and $±(J) = $± \ We also denote Qj = Q n (Eiej Z %)> Qj = Q ± H Qj, and 
Q ± (J) = g ± \g^. Let ( J) = f,©(0 ae$jfla ) , andfli J) = © Qe$±(J) fl Q . Then we have 
the triangular decomposition: 



(J) m m 



(6.i7) = 0^©0r©0 



+ ' 



where 0q J ' ) is the Kac-Moody superalgebra (with an extended Cartan subalgebra) as- 
sociated with the generalized Cartan matrix Aj = (c%)ij e j and the set of odd indices 
jodd _ j n jodd _ |j e j| _ an( j ^(J) ^ reS p_ j s a direct sum of irreducible 

highest weight (resp. lowest weight) modules over (cf. PH). 

To apply our supertrace formula (|2.17|) to the Lie superalgebra q^, we need to 
compute the generalized supercharacters of the homology modules Hk(g^). We con- 
jecture that the 0g J ' ) -module structure of H k (g^P) is determined by Kostant's formula. 
More precisely, let Wj be the subgroup of W generated by the simple reflections rj with 
j G J, and let W(J) = {w G W\ $ w C <& + (J)}, where $ w = {a G $ + | w^a < 0}. 
Thus PVj is the Weyl group of the Kac-Moody superalgebra g^ and W(J) is the set 
of right coset representatives of Wj in W. That is, W = WjW(J). Let us denote by 
$^ = $j n $^ (i = 0, 1) and <&f(J) = <&f \ (i = 0, 1). Then we have the following 
conjecture for the structure of Hk(g_): 

Conjecture: 

(6.18) H k (g^) = Vj(w(p-p)-pf*W, 

wew(J) 
/3e$+(o) 

l(w)+\/3\=k 

where Vj(/i) denotes the irreducible highest weight module over the Kac-Moody su- 
peralgebra 0o' 7 ' ) with highest weight /i. □ 



Remark. The formula ( 6.18|) was first introduced by Kostant for finite dimensional 



simple Lie algebras (El). It was generalized to symmetrizable Kac-Moody algebras 
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ITf , and generalized Kac-Moody algebras But, in general, it is not true 



for Lie superalgebras. Still, we conjecture that Kostant's formula holds for general- 
ized Kac-Moody superalgebras associated with the Borcherds-Cartan matrix which is 
colored by a coloring matrix. 

For each k > 1, let 

(6.19) H ( k J) = Vj(w(p-i3)- P r^\ 

w£W(J) 

/9e$+(o) 

l(w)+\f3\=k 

and define the homology superspace 

oo 

(6.20) # (J) = ^(-l) fc+1 i^ J) = H[ J) e H ( 2 J) © H ( 3 J) 

fe=i 



an alternating direct sum of superspaces. Then, for all g G G and a G we have 
(6.21) 

strfolffM) = (-l) Kw)+m+ HP) M9\Vj(w(p - 0) - p) a ), 

w£W(J) 

/3e$+(o) 

i(«0+|/9|>l 

and the generalized denominator identity can be written as 

J] exp ( -jr ^str(g k \Q a )E ka J = 1 - sch g H^ 



(6.22) 



(-l) l W + We(P) sch g Vj(w(p -0)-p). 



wew(J) 
/3e$+(o) 



Let P(H^) = {reQ-\ str(s|^ J) ) ^ 0} = {n, r 2 , r 3 , • • • }, and t g (i) = stv(g\H^). 
For r G Q~, define the set T^ j \t) of all partitions of r into a sum of 7i's as in (|2.15| ) and 
define the Witt partition function Wg J \r) as in ( |2.16|) . Then, our supertrace formula 
(pT7|) yields: 



Theorem 6.5 Suppose that a group G acts on the generalized Kac-Moody superalgebra 
q = Q(A,m,C) by Lie superalgebra automorphisms of g± preserving the root space 
decomposition. Assume further that str[g\Q a ) = str(g\Q^ a ) for all g G G and a G Q + . 
Let J be a finite subset of I re . Then, for all g G G and a G $ _ (J) ; we have 

str[g\ Qa ) = Y^- d MW i g J i ) (°) 
(6-23) ^ f| , 

d\a sGT( J )(a/d) 
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where /i is the classical Mobius function, and, for a positive integer d, d\ot denotes 
a = dr for some r G Q_ ; in which case a/d = r. □ 



We would like to remind the readers that our supertrace formula ( |6.23j ) has been 
derived under the assumption that Kostant's formula holds for generalized Kac- Moody 
superalgebras. However, in some special cases, we can derive the supertrace formula 
for generalized Kac- Moody superalgebras wihtout assuming Kostant's formula. 

For example, suppose that the Borcherds-Cartan matrix A satisfies: (i) the set 
I re is finite, (ii) aij ^ for all i,j G P m . Then it is known that the Lie su- 
peralgebra gL J) = QeA -(j) 0« ( res P- 0+ J) = Qe A+(j)0a) is isomorphic to the 
free Lie superalgebra generated by the superspace V = je/ im Vj(— aj)® mj (resp. 
^* = 0je/™ a j)® mj )i where Vj(fi) (resp. Vj(fi)) denotes the irreducible highest 
weight (resp. lowest weight) module over the Kac-Moody superalgebra with high- 
est weight fi (resp. lowest weight — /x) (cf. PD|). Hence, for any g G G, the generalized 
denominator identity of the Lie superalgebra is the same as 



(6.24) ae*-(J) 



J] exp(-J2l^(g k \g a )E k A 

$~(J) V fc=l / 



1 - schgV 



1 - rrijSchgVji-aj). 



Hence, by our supertrace formula ( |2.17|) , we obtain 
(6.25) stv(g\ Qa ) = -Ad) £ II V(0*S 



d ^ s' 

!]q seT( J )(a/d) 



where t ff (i) = str^jKj = E,- e /"™ m iS tr (5 , lK/(-aj)rJ for z = 1,2,3, 



7 Monstrous Lie Superalgebras 

Let F(q) = ^^^L_ 1 f{n)q n = q^ 1 + f(l)q + /(2)g 2 + • • • be a normalized g-series such 
that /(— 1) = 1, /(0) = 0, and f(n) G Z for all n > 1. We say that F(q) is convergent 
if it converges for all q = e 2mT , Imr > 0. Observe that, for each m > 1, there exists a 
unique polynomial -P TO (t) G Z[t] such that 

P m (F) = g- m mod 9 Z[[g]]. 

A convergent normalized g-series F(q) = Yl™=-i fi n ) ( i n = Q~ l + /(I)? + /(2)g 2 + • • • 
is called replicable if for all m > and for all a|m, there exist normalized g-series 
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F^(q) = J2n=-if (a) (n)q n = q" 1 + f {a) {l)q + f {a) {2)q 2 + ■ ■ ■ with = 1, 

/(°)(0) = 0, and f {a) (n) G Z for all ra > 1 such that 



(7.1) FW = F, and £ F« fep) = P, 



in 



ad=m 
0<b<d 



where q = e T , Imr > 



In ||40|1 , the replicable functions were characterized in terms of product identities. 



More precisely, a normalized g-series F(g) = q^ 1 + f (l)q + f (2)q 2 + ■ ■ ■ = X^°=-i f( n )q n 
is replicable if and only if for all k > 1, there exist normalized g-series F^(g) = 
Y^=-i f^ k \n)q n satisfying the product identity 

oo / oo .. \ oo 

(7.2) II ex P - E ^f (k) (mn)p km q kn J = 1 - ^ f(i +j - l)pV- 

m,n=l \ k=l / *,i=l 

Now, we will consider the generalized Kac-Moody superalgebras associated with 
the normalized g-series. We take / = { — 1} U {1,2,3, •••} to be the index set, and 
let A = (— (i + be the Borcherds-Cartan matrix of the Monster Lie algebra 

(||). Consider a normalized g-series F(q) = Y^L-i f( n )q n sucn that /(— 1) = 1, 
/(0) = 0, and f(n) G Z for all n > 1. We define the charge of the matrix A to be 
m = (|/(i)| : i G I), and choose a coloring matrix C = (&ij)i,jel such that 6 a = 1 if 
f(i) > and 9u = —1 if f(i) < 0. That is, an index % G / is even if /(z) > and odd 
if f(i) < 0. 

The generalized Kac-Moody superalgebra £(F) = Q(A,m,C) associated with the 
above data is called the Monstrous Lie superalgebra associated with the normalized 
g-series F(q) = Y^=-if( n )Q n - For example, the Monstrous Lie superalgebra as- 
sociated with the elliptic modular function J(q) = j(q) — 744 is the Monster Lie 
algebra, and the Monstrous Lie superalgebras associated with the Thompson series 
Tg(q) = Y^=-x c gi n )q n are the Monstrous Lie superalgebras given in ||. 

Let £(F) be the Monstrous Lie superalgebra associated with a normalized g-series 
F(q) = S^L-i f{ n )q n - By our choice of the charge and the coloring matrix, we see 
that a_i is the only real even simple root, aj (i > 1) is an imaginary even simple root 
with multiplicity f(i) if f(i) > 0, and an imaginary odd simple root with multiplicity 
—f(i) if f(i) < 0. If we identify the simple roots a„i with (1, —1) G Iii,i and with 
(1, i) G (* > 1); then the Monstrous Lie superalgebra £(F) becomes a / /^-graded 
Lie superalgebra, and we have 



£(F) + = £(F) (m , n) , = *W 

m>0 m>0 
nGZ nGZ 



(— m,n) ■ 



Suppose that a group G acts on £(F) by Lie superalgebra automorphisms of £(F)± pre- 
serving the 7/i^-gradation. Assume further that str(g|£(F)( m n )) = str(<7|£(F)(_ m) _ n )) 
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for all g E G, m,n > 0, and 



g-f-x = f-i, g-e_ l = e_ 1 for all g £ G. 

We will apply our supertrace formula ( |6.23|) (actually ( |6.25| )) to this setting. 

Let V-! = £(_i,i) = C/_i, Vi = £(F)(-i,_o = C/ i( i © • • • © C/i,| /(i) | for i > 1, and 
^ = ©~ _x K- For g E G and i E I — {—1} U {1, 2, 3, • • • }, define / 9 (z) = str(</|V5). 
Then V = ©^° = _ 1 V n is a Z-graded representation of the group G such that sch(V) = 
F(q) = J2n=-i /(")?" ^d Bch,(V) = F,(g) = ££=-i / g (n)<f. 

Consider a basis of consisting of the eigenvectors v^k of g E G with eigenvalues 
\,k (k — 1, • • • , |/(0D- Then we have 

1/(01 

/ g (i)=str(^|^)=V(l,i)^A i|fe . 

fc=i 

To apply our supertrace formula ( |6.23| ), take J = { — 1} and consider the corresponding 
triangular decomposition 

2(F) = £(F)L J) © £(F) ( J) © £(F)^ J) . 

Then £(F) ( J) = (e_i, + f) = s/(2,C) + f), and the subalgebra £(F)<l J) (resp. 

£(F)^) is isomorphic to the free Lie superalgebra generated by the superspace = 
e,=iK/(-« l )®' / «l (resp. H( J > = @ZiVj(-<*i) m % where Vj(- ai ) and V}(-a % ) 
are the i-dimensional irreducible representations of s/(2, C) (since — aj(/i_i) = z — 1). 
We will concentrate on the Lie superalgebra Z(F)^P ■ 

Since the weights of Vj(—ai) are — = (—1, —i), —ai — a_i = (—2, — z + 1), • • • , 
— on — (i — l)o!-i = (— i, —1), we have 

P(HW) = {(-i,-j)\i,j GZ >0 }. 

Note that each /j^ generates an i-dimensional irreducible representation of the Lie 
algebra si (2, C) generated by e_i, h-i, and hence so does each v^k (i > l,k — 
1, 2, • • • , \f(i)\). Thus we have 

l/(<+i-l)| 

fc=l 
l/(<+i-i)| 

= CCo^-i) 4 - 1 ^-!.*). 
fc=i 

It follows that 

5 • (adf-xY^ivi+j^k) = (ad(g ■ f-i)) l ~ l (g • v^-i,*) 

= \i+j-i,k(adf-iy 1 (t> i+: ,-_i i fc), 
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which yields 



str^g^) = 1>{l,i + j -I) ^-i,k = f g (i + j - 1) 

fc=l 



for all g E G, i,j £ Z >0 . Hence the generalized denominator identity for g G G of the 
Lie superalgebra £(F)_ is equal to 



(7.3) 



II ex P (-E^Wk^!*" 

n,n=l \ fe=l / 

oo 

= l-^/ g (z+j-l)pV. 



For k,l > 0, we have 

T {J \k,l) = T(k,l) = {s= (sij)ij>i| s y G Z> , = (M)}> 

the set of all partitions of (k, I) into a sum of ordered pairs of positive integers, and 
the Witt partition function Wg J \k, I) is given by 



(J) \Sij 



Si 

seT(k,l) 



si 

seT(fc.Z) 



rn 



Therefore, our supertrace formula ( |6.23|) yields: 

Proposition 7.1 Let £(F) = @( m n)eih i (m,n) be the Monstrous Lie superalgeb 
associated with a normalized q-series F(q) = Yl™=-i fi n )Q n an d ^ V — ®^L_i V n = 
£(F)(-i,-n)- Suppose that a group G acts on £(F) by Lie superalgebra automor- 
phisms of £(F)± preserving the 1 1^%- gradation. Assume further that s£r(g|£(F)( m>n )) = 
sir(g|£(F)(_ m _ n )) for all g G G, m,n > 0, and 

9 ■ f-i = f-i, 9-e-i = e_i for all g G G. 
Then, for all g G G and m,n > 0, we have 

str(g\£(F)( m;n )) = str(g\£(F) ( _ m _ n) ) 

{TA) - E \m E ^nw+i-')*. 

d\(m,n) »€T(f ,5) 

where f g (i) = str(g\Vi) = str(g\£(-.i-i)) for all g G G and i = 1, 2, 3, ■ • • . □ 
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Corollary 7.2 Suppose that the generalized supercharacters F g (q) = Y^=-i fgi n )Q n 
are replicable with the replicates Fg ^ (a > 1) for all g £ G. Then we have 

(7.5) S tr(g\2(F) im>n) ) = str{g\Z{F) { „ m _ n) ) = £ ^{d)f$ ( 



a 2 d 2 

ad\(m,n) 



Proof. Since the generalized supercharacter F g (q) is replicable, by (|7. 1| ) , the generalized 
denominator identity for g 6 G of the Lie superalgebra £(.F)_ is equal to 



oo / oo 1 

II exp -^-str(/|i:(F) ( _ mi _ n) )/V n 

n,n=l \ fc=l 

oo 

= l-X)/,(i + j-l)pV 

oo / oo 

= II ex p -E-4 a W)p ar v 



a 

m,n=l \ a=l 



By taking the logarithm, we have 

oo / oo 1 

log J] exp -X;^tr(/|£(F) ( _ m> _ ri) )p fe V n 

m,n=l \ A;=l 

oo oo 1 

= - E Efc str ^'i £ ( F )(-"))^ fcn 

m,n=l fc=l 
oo 

= TjStr(^|£(F) ( _ f ,_ f) )p m g^ 

m,n=l fc>0 
/c|(m,n) 



and 



oo / oo 1 

log J] exp(-£-/W(mn)p fl *V 

a=l 

oo oo 1 

- E E-4 a) ( mn )^ am ^ 



a 

m,n=l \ a=l 



m,n=l a=l 



oo 1 

e e k ( ° 



a 3 \ a 2 

m,n=l a>0 
a| (m,n) 



P Q 



It follows that 



E H^-t-p' - E (=£) ■ 



fc>0 a>0 
fc|(m,n) o|(m,n) 

Therefore, by Mobius inversion, we obtain 

str(9|£(-F)(-m,-»)) = £ ^"W^O- ° 

od| (m,n) 
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8 Orbit Lie Superalgebras for the Dynkin Diagram 
Automorphisms of Borcherds-Cartan Data 



In the following two sections, we will discuss the applications of the generalized de- 
nominator identities for Dynkin diagram automorphisms of generalized Kac-Moody 
superalgebras. The notion of orbit Lie superalgebras will play an important role in 
our discussion. Most of our result is a generalization of the corresonding result on 
Kac-Moody algebras and generalized Kac-Moody algebras obtained in [[L3[] and |14 . 



Let A = (aij)ij(zj be a Borcherds-Cartan matrix of charge m = (m.j| i £ I) with a 
coloring matrix C = (%)ijei; an d let q = q(A, m, C) be the corresponding generalized 
Kac-Moody superalgebra. 



Definition 8.1 A bijection o : I — > / is called a Dynkin diagram automorphism of the 
Borcherds-Cartan data (A,m,C) if it satisfies: (i) = o-a-UYa-U) f° r ai l hj Ii (h) 
a(I even ) C I even , a(I odd ) C I odd , (iii) m a{i) = m, for all i G J. 

In this paper, we will assume that the Dynkin diagram automorphism has finite order 
N. 

Given a Dynkin diagram automorphism a of a Borcherds-Cartan data (A,m, C), 
we extend it to a Lie superalgebra automorphism of g = g(A, m, C) by defining 

cr(e ik ) = e a (j) )k for i G /, 1 < fc < m i3 

(8.1) cr(/ijfe) = /<r(i),fc for i £ 1,1 < k < m h 

v{hi) = h^i), a(di) = d ff (i) for i £ I. 

It is straightforward to check that a is indeed a Lie superalgebra automorphism of q. 
Let U = U(q) be the universal enveloping algebra of q. Then the Dynkin diagram 
automorphism a of q can be extended to an algebra automorphism of U(q) by 

a(xi ■ ■ ■ x k ) = a(xi) ■ ■ ■ a(x k ) for x { e Q, 

which we will also denote by a. 

Moreover, the Dynkin diagram automorphism a induces an automorphism a* of f)* 
defined by 

(8.2) o-*(A)(/i) = \(<r~ l (h)) for A G f)*, h G f). 

In particular, for all z G /, we have cr*(ai) = awi). Since a has order N, a* also has 
order N, and we have an eigenspace decomposition 

N-l 

r = 0(D*) (fc) , 

fc=0 
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where (f)*)« = {A G ff| a* (A) = e^A}. The elements of (f)*)(°> = {A G [fl <r*(A) = 
A} are called symmetric. 

Now, we will construct a generalized Kac- Moody superalgebra g = fl(cr), called the 
orbit Lie superalgebra, associated with the Dynkin diagram automorphism a of g. Let 
a be a Dynkin diagram automorphism of the Borcherds-Cartan data (A,m, C). Since a 
permutes the elements of 7, 7 is decomposed into a disjoint union of cr-orbits: 7 = |_|[«], 
where [z] = {a k {i)\ k — 0, 1, • • • , TVj — 1}. Let TVj = |[z]| denote the number of elements 
in the er-orbit [i]. For each i G 7, let i be the smallest element of the a-orbit of i, and 
let I = {i\ i G 7} be the set of such representatives. Set T re = 7 P 7 re , f m = 7 P 7 im , 

jeven 7 P J even gj^d J"^ 7 Pi 

Let A = v4(cr) = (o|j)jj 6 j be the square matrix indexed by 7 whose entries are 
defined by 

(8.3) au = ej ^ a i,oHi) for e I> 



k=0 



where 6j = 1 — X)fc=i 1 a j,o- k (j)- Note that = e^y) for all j G 7. 
Let 

(8.4) 7 = { i G 7 | e, < 2 for z G 7 re , e, = 1 for z G I im }. 

Since aji = 6i(a# + 1 — 6,), the condition on 7 is equivalent to aj j = an for all i G 7. 
(The first half of the condition ( |S.4| ) was introduced in |13| and was referred to as the 



linking condition.) Also, we set A = A(a) = (%)jj e j, 7 re = 7 P J re , P m = I P 7 im , 

jeven / p J even gj^d 7 Pi 

One can easily verify that A is a symmetrizable Borcherds-Cartan matrix. For 
instance, let = TV^s, (z G 7) and let 79 = dia^^l i G 7), then 79^4 is symmetric. 

We define the charge m = (fnq\ i G 7) and the coloring matrix C = (^ij)j j e f of A 

by 

(8.5) rhi = m h 0? j = % for z',jG 7. 

Consider the generalized Kac- Moody superalgebra g = g(A, m, C) associated with the 
Borcherds-Cartan data (A, m, (7). We denote by ej fe , /i )fc , ^, dj (i <E I,k = 1,2, - ■ ■ , m,) 
the generators of Lie superalgebra g, and by f) = ^©- g j C/ij j © (® i e 7 CdH the Cartan 

subalgebra of g. We also denote by LT = {aj\ z G 7} the set of simple roots of g and by $ 
the set of roots of g. The orbit Lie superalgebra g = g(<r) is defined to be the subalgebra 
of g generated by e~ ijk , f^ k ,(i G 7, k = 1, 2, • • • , m,) a\ (i G 7, k = 1, 2, • • • , m*). We 
denote by II = {cq| z G 7} the set of simple roots of g and by $ the set of roots of g. 

As in the case of g = g(A, m, C), there is a symmetric bilinear form on f)* satisfying 
(A|6q) = \(sjhA for A G f)*, z G 7 (see (|6.2|) ). Define a C-linear isomorphism : 
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(ff) (0) - Jf by 

(g6) 0(A)(/T;) = A(/^), 

0(A)(5) ^e-U^) for AG (f)*) {0) , ie/. 

Then we obtain the following lemma which will be used in the next section. 
Lemma 8.2 For X, fi G (f)*) (0) , we have 

(A|/i) = (0(AM^). 

Equivalently, for X, G f)* ; we /iawe 

(A|/2) = (0" 1 (A)|0- 1 (/2)). 

Proof. First, observe that, _1 (ciq) = £i E^o 1 ^'!') ^ or * e ^- Hence for A G f)*, we 
obtain 

(A|a-) = X(sihi) = sj\(hi), 

and 

Ni-l Ni-1 ^ 

(0 _1 (A)|0 _1 (a ? )) = (0 _1 (A)|ei a CT fc W ) = Q ^ (^Wl ^)) 

fc=0 fc=0 

iVj— 1 

= Cj -1 (A)(s CT fc W /i CT fc(i)) = e^i0- 1 (A)(s i /ij) 

fc=0 

= e i N i s i (t)~ 1 (X)(hi) = Si6iNiX(hi), 
which proves our claim. □ 

Observe that if = 1, the a-orbit of i in the Dynkin diagram of g is the disjoint 
union of N { copies of Ai diagram or B(0, 1) diagram. If q = 2, then N { is even and 
the fj-orbit of % in the Dynkin diagram of is the disjoint union of copies of A 2 
diagram. 

Next, we will show that the Weyl group W of g is isomorphic to a subgroup of W. 
We may assume that W acts on h* since q is a subalgebra of g. For each i G J re , we 
denote by wj the simple reflection in W and define 



(8.7) 



nNi/2-1 -r o 



where Wj is simple reflection in (i e J re ). Since a ij(T ku\ = for all /c ^ (mod iVj) 
if €i — 1 and a i ^ a N i /2^ = — 1 if e$ = 2, is well-defined. Let be the subgroup of 
generated by wj's (i G J re ). 



40 



Proposition 8.3 For alii G I re , we have: 

(a) rD? = l. 

(b) a*wq = wqa* . In particular, (f)*)^ is invariant under w^. 

(c) For each A G (f)*) < -°' ) , we have 

Ni-l 

Wi(X) = A - eiA(/ii) ^ a CT fc(i)- 

(d) For eac/i simple reflection wj G W 7 , we have wj = (j) -1 ^^. 
Proof. The proofs of (a), (b) and (c) are straightforward. 

For (d), let A G (f)*) (0) . Then we have 

r'wi 0(A) = ^^(A) - (f>(X)(hj)oq) 
= r x (0(A) - A(^)S?) 
= A-A(W 1 (^) 

JVi-l 

= A - X(hi)6i a CT fe (i) 

fc=0 

= tw?(A), 

which proves our assertion. □ 

Proposition 8.4 T/ie VKeyZ group W of the orbit Lie superalgebra g isomorphic to 
the subgroup WofW under the map Wj i— > ^ (i G J re ). 

Proo/. See |TJ or 0. □ 



Remark. The original proof of |8.4] in |T3] consists of a somewhat lengthy calcu- 
lations. In ||14|| , the proof was simplified and it was also proved that W = { w G 
W | wa* = a*w }. 

Under the above identification of W with W, we define the sign function e : W — > 
{±1} of W by 

e(w) = e^wcj)^ 1 ) = e^" 1 {w)) for w G W, 

where e denotes the sign function of the Weyl group W of the orbit Lie superalgebra 
= fl(o"). Note that, even though W is a subgroup of W, we use the sign function of 
W, not the sign function of W . 
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9 Generalized Characters of Highest Weight Mod- 
ules for Dynkin Diagram Automorphisms 

Let A G P + be a dominant integral weight and let M(A) be the Verma module with 
highest weight A. By construction of M(A), we have 

M(A) = U(b)/R(A), 

where -R(A) is the left ideal of U(q) generated by the elements (i G /, k = 1, • • • , m,) 
and h-A(h)l(heij). By Q and we see that a(i?(A)) = R(a*(A)). Hence 

the Dynkin diagram automorphism a induces a C-linear isomorphism 

(9.1) a : M(A) -> M(cx*(A)) 

between Verma modules. Note that we have a(M(A)\) = M(a*(A)) a *^ for all A < A. 
Similarly, let V(A) be the irreducible highest weight module with highest weight 



A. Then by Corollary |6.4| , the Dynkin diagram automorphism induces a C-linear 
isomorphism 

(9.2) a : V(A) -»■ V(a*(A)) 

between irreducible highest weight modules, and we have a(V(A)\) = V r (cr*(A)) cr .( A ) 
for all A < A. 

In particular, if A G (f)*)^°\ i.e., if ex* (A) = A, then the Dynkin diagram automor- 
phism a induces C-linear automorphisms on M(A) and V(A), and we can consider the 
generalized characters for a: 



ch CT M(A) = ^tr(a|M(A) A )e > 



(9-3) ^ 

ch CT \/(A) = ^tr( ( x|y(A) A )e A . 

A<A 

Note that we have only to take the sum over the weights A such that cr*(A) = A. 

Recall that the automorphism a* permutes the roots of g in such a way that we 
have 

(7 *($±) = $±, a *($±) = $± c7*($±) = $± 

Thus $ ± is a disjoint union of <r*-orbits of the elements in $ ± . Let denote the set 
of representatives of cr*-orbits in $ ± and set 

s£ = s ± n$ , st = S ± n& 1 . 

For each root a G $, let N a denote the number of elements in a*-orbit of a, and define 

N a -1 N a -1 

(9.4) (a) = °* k (a) and g (a) = @ Qo**{aY 

k=0 k=0 
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Then, by the Poincare-Birkhoff-Witt Theorem, we have 

£/(£)*) ^(so^A^f), 

where 

S(flo ) = (g) S( Q(a) ), and A( fl f) = (g) A( 0(/3) ) 

ae5± /3eS± 

as C-vector spaces. Hence the generalized (super) character of the Verma module M(0) 
for the Dynkin diagram automorphism a is given by 



ch CT M(0) = sch a M(0) = ^tr(a|M(0) A )e A 

A<0 

= n (E tr ^i 5m (0(-))) e_m(a)/jVQ ) 
x n fe tr ^i An (s(-^))) e_ " ( " )/jV5 ) 

= II ex P (E ^r(^| 0( _ a) )e-^) 

ae,S+ \m>l J 

x nexpf-^t^trK| 0( _, ) )e-^ 



n 

n>l 



-m(a)/N a 



II exp(^lstrK| 0( _ Q) )E- 

«G5+ \m>l 

n ex p fE^ str (- m i0(-))^ m(a)/jVQ ) 

\m>l a / 



Note that if p is a Weyl vector of the generalized Kac-Moody superalgebra q, then 
4>(p) is a Weyl vector of the orbit Lie superalgebra g. 

Proposition 9.1 For all w G W , we have 

(9.5) w (e' p ch a M(0)) = e(w)e~ p ch a M(<d). 

Proof. It suffices to prove our assertion for the simple reflections wi G W (i G I re ). 
For each % G I re , let = exp(e-) exp(— //) exp(e-), where the elements e-, // and /i- are 
defined by 



e' t = e t , f! = fi, K = h t iiiel even , 
e'^We^el fl = \[fi, fi], K = \hi if i G 1° 
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Then for each i £ I, the elements e^, f-, and h! i generate the subalgebra of g isomorphic 
to sl(2, C), and by the same argument in pD |, we can verify that G Aut(g) and 

r i(fla) = 0Wi(a)- 

We first consider the case when = 1. (Recall that, in this case, aj i(T (j) = 0.) 
Suppose a ^ aj and a 7^ 2«j. If we put f\ = Ylk^o 1 r o- k (i)> then since a^o-fcm = 0, we 
S et r ^'(t) r aHi) = V(i)V'w !t follows that 

str(a m |g ( _ Q )) = str(f^ a m f- _1 |g ( _^ a) ) = str(cr m |g ( _^ Q) ). 
If a = otj and i e / ew;n then, since 



str(a m | S( _ a!) ) 

we have 



Ni if m = (modiV;), 
otherwise, 



exp I J2 ^sti(a m \ g{ . ai) )E- m ^ N A 

\rn=l J 



(1 _ 



and 

wj(e- p (l - .ET^)- 1 ) = e - p+(ai) (l - E^Y 1 = -e~ p (l - E' {a ^)-\ 
Similarly, if a = cti or a = 2cti for i e then 



and 



exp ( jr ^stT(a m \ d ^ at) )E- m ^y N ') 

\m=l / 
( °° 1 

x exp ^ -str( ( x m |g ( _ 2ai) ) J E;- m ^)^ 

\m=l 

= (1 (1 -E- (2a ' y )-\ 



= e -p+( a i)(\ - E^ ai) ){\ - E^^y 1 

= -e- p+{a '\l + e (ai) )(l - e^) -1 
= -e~ p {l - E~^)(l - E-P"*))- 1 . 



\a N i/ 2 (i) 



Next, we consider the case when e» = 2. In this case, we have a i 
which implies that % G l even . Suppose a/aj and a 7^ + a a Nj2^ for any k = a l {i) 
(1 < I < Ni/2). Set fj = Yl^li 2 " 1 r u i {i) r al +N l /2 {i) r al{i) . Since a kta N l/2{k) = -1 implies 
r k r a Nj2 {k) r k = r a Nj2 {k) r k r u N l /2 {k) (see, for example, @), we have or- = fja and 



str(a m |0(_ Q) ) = str(r-cr m r. 1 \Q { -u,. a )) = str(cr m |g ( _^ Q) ). 
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If a = ctk or a = + a a Nj2^ for some k = a l {i) (1 < I < (i.e. (a) = (a^)), 

then 



exp 



J -str( ( T m |g ( _ Qi) )E- m(Q!)/ ^ ) = (1 - E-^Y 1 

\ m =l m ) 



exp (E|str( ff -| 8( _ e ,,_ £ ,^ i/2(ii) )£:- 2 ™<« + «.''.'=<.,'« ] = (! + £-(«))-., 

\TO=1 



and 



= -e~ p {l - E { - a ^)-\l + S^) -1 , 
which completes the proof of our assertion. □ 

Proposition 9.2 For A G P + and w 6 W C W 7 , we /iai>e 

(9.6) w(c/i CT F(A)) = cha-V (A) . 

Proof. It suffices to prove Q9.6Q for w = (i E f re ). Let 7r : g — > g/(V^(A)) denote the 
representation of g on V(A) and define 

-7T _ J llfc=o r CT fc (i) if e * = 

[lL=o ^(ijv+^/a^r,!^) ife, = 2. 

By the same argument in the preceding proposition, we have err? = ffa. It follows 
that 

tr(a|y(A) A )=tr( ( x|y(A)^ A ), 
which proves our assertion. □ 

Now, we are in a position to state and prove the main result of this section. 

Theorem 9.3 Let a be a Dynkin diagram automorphism of a generalized Kac- Moody 
superalgebra g and let g = g(cr) be the corresponding orbit Lie superalgebra. Let 4> '■ 
(f)*)(°) — » fj* be the C-linear isomorphism given by (|8.6|) , and extend it to a C-linear 
map : C[[(f)*) (0) ]] -> C[[f)*]] 6y 0(e A ) = e^ A ) /or A G (f)*) (0) . T/ien, /or a symmetric 
dominant integral weight A G P + PI (h*)*- -*, we have 

(9.7) 0(c/i CT y(A)) = cM>(0(A)), 

where V(<f>(A)) is the irreducible highest weight Q-module with highest weight 0(A). 
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Proof. Using the properties of Casimir operator given in Proposition |6.2| , the same 
argument in |H| yields 

(9.8) cKV(A)= Yl c x cKM(X). 

A<A 
|A+p| 2 = |A+p| 2 

Since ch CT M(A) = e A ch ff M(0), by (0), we have 

(9.9) e p RAV{h)= Yl CAeA+P ' 



|A+p| 2 =|A+p| 2 



where 



R a = (ch CT M(0))- 1 = J] exp ( - ^Mv m \9(- a ))E~ m{a)/Na ) ■ 

ae$+ \ m>l m a / 

We may assume that the sum in the right-hand side of ( |9.9j ) is taken over the weights 
A which are invariant under a*. 

If w(\ + p) = t + p, then c T = e{w)c\. Choose a weight r such that c T ^ 0. Then 
w{t + p) < A + p for all w G W. Let A be an element of the set {w{t + p) — p\ w G W } 
such that ht(A — A) is minimal. Note that (A + p|«j) > for all % G J re \J re by definition 
of /. If (A + p\oti) < for some i G I re , then -u^(A + p) = A + p — e«(A + p){h.j){oti) 
and ht(A — u^A)) < ht(A — A), which is a contradiction. Hence, (A + p\oti) > for all 
i G J re . Moreover, since |A + p\ 2 = |A + p\ 2 , we have 

(9.10) = |A + p\ 2 - |A + p\ 2 = (A|A - A) + (A + 2p\A - A), 

where A — A = J2iei n i a i ues m 
If % G J re , then 

(A + 2p| ai ) = (A + p|ai) + (p\an) > 0. 
If i G P m and > 1, then 

(A + 2p|«j) = (A + ttilcti) 

= (A - njCCj - (rii - l)ai\a>i) 
iei\{i} 

= (A|«j) - 2j rij («j | oii) - (rij - l)(ai | ttj) > 0. 

It follows that rij = for all i G J re . That is, A = A — Yliei im n i a i- dFrom ( |9.10 ), for 
k G P m with rik > 1, we have 

(A + 2p|a fc ) = (A + a fe |a fc ) = (A|a fc ) - n-(aj|a fe ) = 0, 
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where v! i = rij for i ^ k and n' k 



rik — 1. Hence we have 



(AM 

(QfilcKfc) 
(afeldifc) 



0, 

if Hi > 1 and i ^ k, 
if ra fc > 2, 



and /c lies in P m . Combining the above conditions and the fact that rii = n ff (,) for all 
i G I, we conclude that A has the form 



A = A - ^2 n i( a *)' 



where 



(A|(oi)) 

((«<) !(«<)) 



if % j and nj, m > 1, 
if nj > 1, 
if n; > 2. 



Let A, A' be two such elements of the above form. Suppose that there exists a w G W 
such that w(A + p) = X' + p. Then, if A G P + , we have w(X) < A, since w(X) is also 
a weight of the g- module V(A). Furthermore, by induction on the length of w, we can 
verify w(p) < p. Hence we obtain w(X+p) < X + p and w(X + p) = X+p— Xl a e*+. re n « a 
with n a > 0. On the other hand, if w(A + p) = X' + p, we must have n a = for all 
a G $ +,re . Hence w(A + p) = X + p, which implies A = A'. 

Let $+(A) be the set defined in (|6U2p and let $+(A)(°) = $+(A) n (f)*) (0) be the set 
of elements Y^iel im n l( a i) sucn that = if nj,nj > 1 for i ^ j, (A|(o;j)) = 

if nj > 1, and ((aj)|(o!j)) = if rq > 2. So ( |9.9D can be written as follows: 



Now, it remains to determine ca_/? for (3 G . First, we observe that ch CT V(A) 
contains no term of the form e A ~^e-r im where (A|«j) = for all i such that rti > 1. 
So, in order to determine c^-p, we have only to compute the coefficient of e A_/3 in R a . 
Note that = implies that a, + a.,- G" 



e^ CT ch (T V(A) = Yl 



w(A-/3+p) 



w<EW /3e#+(A)C°) 



For each i G /* m , we have 




For p = Yjiefim n i( a i)i using the formal power series expansions 
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and 

we obtain 

ca-p 



(l+x)- = ±( n + k - 1 )(-l?*\ 



k=0 



n t) (_1)n? n ( m ' + ^ 1 \-ir = {-iy* w Hm)- 



Je.ve.ri i(Zjodd 

When A = 0, we get 

iCGW Ae*+(o)(°) 

Since 0((a»)) = for i e J im , by Lemma |S.2| , we have ((ai)\(aj)) = if and only if 
(cq|aj) = and (A|(«i)) = if and only if (0(A) \cq) = 0. It follows that 0($+(A)(°>) = 
$ + (0(A)) C f)*. Therefore we obtain 



0(ch CT V(A)) 



S^gggg) Wo) c A -^ A -^)-^ 

l^w£W b \ W ) l^PE®+(<p(A)) c A-4>-i(j3) c 

E^w^) E^+( ) c^-i (9) e-(-7+p)-P 

E«^WE^(o)(-l)^(T)e^- p 
chV r (0(A)). 



□ 



-Ro- = e(iu) c_ 7 e" 



Corollary 9.4 

0(^) = n (i - J B- a ) sdim5 « : 

(9.11) ae 3>+ 

(f)(ch a M(A)) = chM((j)(A)). 

Proof. In the proof of Theorem 9.3, we have shown that 

w€W 7G<J>+(0)(°) 

Hence we have 

<K^) =]>>>) E c_ 7 e^ 7+p) ^ = E £ H E c_,- 1{7) e^ + ^ 

w&W 7G<I>+(0)( ) w&W 7Gi+(0) 

= J2e(w) (-l)^£(7)e* ( ^ +p) - p = JJ (1 -E- Q ) sdim§Q . 

lieiy 7e4+(o) ae3>+ 
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Since ch cr M(A) = e A i? (T 1 , we obtain 

0(ch CT M(A)) = e^ A) ] [ (1 - £-«)-sdimfl Q = c hM(0(A)). 



Set Q = ®- 6 yZ(ai/ejiVj), and define a Z-linear map : Q — > Q by 

= eq/e{Ni for i E I, k = 0, 1, • • • , iVj — 1. 
Then is an extension of 0|Q n ([f)(°) to From now on, we will also use for 0. 



□ 



For each j3 £ Q, define the subspace gui of g to be 

S[/3] = 0«- 
ag</.-l(/3) 

Clearly, [0[ a ], fM] C Q[ a +/3], and hence g becomes a Q-graded Lie superalgebra with 
the coloring of Q induced by that of Q. Note that the Dynkin diagram automorphism 
a preserves the Q-gradation of g. With respect to this gradation, the generalized 
denominator identity for the Dynkin diagram automorphism a of the generalized Kac- 
Moody superalgebra g is the same as 

(9.12) l[ exp I - ^str(a m \ Qhl3] )E- m n = 1 - scKH(g_). 

/3eQ+ V m=1 / 

On the other hand, we have 

/ oo 

m) = n ex p - E ^M* m h- a) )E- m ^v N " 

ae<S>+ \ m=l a 

(9.13) = n ex p f- e ^^ m \^ E - mHa) ) 

«e$+ V m=i a J 
= II ^p(-E^tr(a m | 0M] )E-^j. 

/3GQ+ V m=l / 

Therefore, by combining (|9.12 ) and (|9.13| ) with Corollary |9.4| , we obtain: 



Theorem 9.5 The generalized denominator identity of the Q-graded generalized Kac- 
Moody superalgebra g for the Dynkin diagram automorphism a is the same as the 
denominator identity of the orbit Lie superalgebra g. □ 

Remark. (a) For the Kac-Moody algebras and generalized Kac-Moody algebras, 
Theorem [O] was proved in and |I4| . They also gave a list of Dynkin diagram 
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automorphisms of Kac- Moody algebras g of finite growth and corresponding orbit Lie 
algebras g ([Jj]|). 

(b) Let D a (g) denote the generalized denominator identity of the Q-graded gener- 
alized Kac- Moody superalgebra g for the Dynkin diagram automorphism a and ®(q) 
the denominator identity of the orbit Lie superalgebra g. Then, Theorem |975| implies 

(9.14) D CT (g)=£(g). 

(c) For the affine Kac- Moody superalgebras introduced in [HO], we have 

£> CT (A (2) (0,2n - 1)) = D(S (1) (0,n- 1)) when n > 2 and a has order 2, 

£> CT (C (2) (3)) = D(A (4) (0, 2)) when a has order 2, 
£> CT (C (2) (0,2n + 1)) = £>(A (4) (0,2n)) when n > 2 and a has order 2, 
lD (T (C (2) (0,2n + 2)) = S(B (1) (0,n)) when n > 2 and a has order 2. 

The identity ( |9.14j ) can be rewritten as 

-a\sdim g(<r) Q 



II ex P (- E istr^lfl^)^] = J] (1 - e- 

9GQ+ V m=1 / «6Q+ 



/3eQ+ \ m=i / a6Q^ 

where g(er) = g is the orbit Lie superalgebra corresponding to the Dynkin diagram 
automorphism a of g. Let us consider g(<J d ) for a nonnegative integer d. Then we have 
the following objects as in the case of 0(c): 

(i) /(d), /(d) iVi(d), e*(d), 1(d), 1(d), Q(d) = ©j e?(d) Z(c^/e<(d)iV i (d)) J 

(ii) a Z-linear map <pd '■ Q Q{d), where 4>d{oci) = oq /ei(d)Ni(d) for % e I, fc = 
1, • ■ ■ , Ni(d). 

Applying the above equation to g(cr d ) , we get 

(9.15) 

ex V (-f^^stT(a md \g hP]d )E' m A= (1 - E'^^*, 

P£Q+(d) V m=l / a€ Q+(d) 

where g^ d = ©^g^-i^) g a - Also, if we define a map r of /(d) by r(z) = cr(z), then it 

induces a Z— linear map <p d : Q(d) — > Q such that <p d 4>di.P) = 4>(P) for all (3 E Q. By 
taking d , ( |9.15 ) can be written as 



(9.16) 

J] exp (-^l s tr(a md | flH3] )£-^) = J] (1 - ^^W, 



m 

,3eQ+ \ m=1 / aG^ 

where g(<r d )[ Q ]d = ©^adwiy fl(0" d )/3- By taking the logarithm on both sides, we have 

oo oo 
/3GQ+ m=1 a£Q+ n=1 
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It follows that | 

^ -str (cr dk \g hT] ) = sdimg(a d 



fe>0 a>0 
/3=kr P= a l 



If we put T a d({3) = Ylalp i s< ii m 0( cr<i )[/3/a] d f° r a nonnegative integer d and (3 <E Q, then 
by Mobius inversion 



str(or fc |s[_ a] ; ^ 

(9.17) 



^ ^(d)T adk (a/d) 

d\a 

sdim s( adfc )[°/^i dfe ' 



where /i denotes the classical Mobius function. 

Let g CT be the fixed point subalgebra of g for a Dynkin diagram automorphism a. 
Then it is also graded by Q and by (|9.17|) we get 

Proposition 9.6 For each (3 G Q, 

1 H 1 

(9.18) sdimtfp = — ^ ^2 ^(d) sdimg(a dk ) [f3/ad]d k, 

' ' k=l ad\/3 

Proof. Note that g^ is a representation of a finite cyclic group (a). Hence dimg^ is the 
multiplicity of the trivial representation of (a) in g^ and it is the same as the average 
of traces of r 6 (a) on g^. □ 

Remark. By the similar argument in |J , we can see that g a is a central extension of a 
generalized Kac-Moody superalgebra. However, the corresponding Borcherds-Cartan 
matrix is not known in general. Also, the decomposition of g a with respect to the 
Q— gradation is coarser than its root space decomposition. 



References 

[1] D. Alexander, C. Cummins, J. McKay, C. Simons, Completely replicable functions, 
in Groups, Combinatorics, and Geometry, Durham Symposium, 1990, Lond. Math. 
Soc. Lecture Note Series 165 (1992), Cambridge Univ. Press, 87-98. 

[2] Y. A. Bahturin, A. A. Mikhalev, V. M. Petrogradsky, M. V. Zaicev, Infinite dimen- 
sional Lie superalgebras, Walter de Gruyter, 1992. 

[3] Y. A. Bahturin, Identical relations in Lie algebras , VNU Science Press, Utrecht, 
1987. 



51 



[4] R. E. Borcherds, Vertex algebras, Kac-Moody algebras and the monster, Proc. Natl. 
Acad. Sci. USA 83 (1986), 3068-3071. 

[5] R. E. Borcherds, Generalized Kac-Moody algebras, J. Algebra 115 (1988), 501-512. 

[6] R. E. Borcherds, Monstrous moonshine and monstrous Lie superalgebras, Invent. 
Math. 109 (1992), 405-44. 

[7] R. E. Borcherds, Automorphic forms on O S+ 2,2 0R) and infinite products, Invent. 
Math. 120 (1995), 161-213. 

[8] A. Berele, A. Regev, A Hook Young diagrams with applications to combinatorics 
and to representations of Lie superalgebras, Adv. in Math. 64 (1987), 118-175. 

[9] N. Bourbaki, Lie groups and Lie algebras, Part 1, Hermann, Paris, 1975. 

[10] H. Cartan, S. Eilenberg, Homological algebra, Princeton Mathematics Series, Prince- 
ton University, 1956. 

[11] J. H. Conway, S. Norton, Monstrous moonshine, Bull. Lond. Math. Soc. 11 (1979), 
308-339. 

[12] I. Frenkel, J. Lepowsky, A. Meuerman, Vertex Operator Algebras and the Monster, 
Academin Press, Boston, 1988. 

[13] J. Fuchs, B. Schellekens, C. Schweigert, From Dynkin diagram symmetries to fixed 
point structures, Comm. Math. Phys, 180 (1996), 39-97. 

[14] J. Fuchs, U. Ray, C. Schweigert, Some automorphisms of generalized Kac-Moody 
algebras, J. Algebra 191 (1997), 518-540. 

[15] D. B. Fuks, Cohomology of infinite dimensional Lie algebras, Consultant Bureau, 
New York, 1986. 

[16] W. Fulton, J. Harris, Representation Theory: A First Course, Springer- Verlag, 1991. 

[17] H. Garland, J. Leopwsky, Lie algebra homology and the Macdonald-Kac formulas, 
Invent. Math. 34 (1976), 37-76. 

[18] V. A. Gritsenko, V. V. Nikulin, Siegel automorphic form corrections of some 
Lorentzian Kac-Moody Lie algebras, Amer. J. Math. 119 (1997), 181-224. 

[19] V. A. Gritsenko, V. V. Nikulin, Automorphic forms and Lorentzian Kac-Moody 
algebras I, Inter. J. Math. 9 (1998), 153-199. 

[20] V. A. Gritsenko, V. V. Nikulin, Automorphic forms and Lorentzian Kac-Moody 
algebras II, Inter. J. Math. 9 (1998), 201-275. 

[21] K. Harada, Modular functions related to the Monster, Lectures given at the confer- 
ence Moonshine and Vertex Operator Algebra, RIMS, Kyoto Univ., 1994. 

[22] J. A. Harvey, G. Moore, Algebras, BPS states, and strings, Nuclear Phys. B463 
(1996), 315-368. 

[23] J. Hong, J.-H. Kwon, Decomposition of free Lie algebras into irreducible components, 
J. Algebra. 197 (1997), 127-145. 



52 



[24] Y.-J. Im, Defining relations of generalized Kac-Moody superalgebras, M. S. thesis, 
Seoul National University (1998). 

[25] E. Jurisich, Generalized Kac-Moody Lie algebras, free Lie algebras and the structure 
of the Monster Lie algebra, J. Pure and Applied Algebra 126 (1998), 233-266. 

[26] E. Jurisich, J. Lepowsky, R. L. Wilson, Realizations of the Monster Lie algebra, 
Selecta Mathematica, New Series 1 (1995), 129-161. 

[27] V. G. Kac, Simple irreducible graded Lie algebras of finite growth, Math. USSR- 
Izvestija 2 (1968), 1271-1311. 

[28] V. G. Kac, Infinite- dimensional Lie algebras and Dedekind's rj-function, Funct. 
Anal. Appl. 8 (1974), 68-70. 

[29] V. G. Kac, Lie super algebras, Adv. in Math. 26 (1977), 8-96. 

[30] V. G. Kac, Infinite dimensional algebras, Dedekind's rj-Function, Classical Mobius 
function and the Very strange formula, Adv. in Math. 30 (1978), 85-136. 

[31] V. G. Kac, Infinite Dimensional Lie Algebras, Cambridge Univ. Press, 3rd ed., 1990. 

[32] V. G. Kac, S.-J. Kang, Trace formula for graded Lie algebras and Monstrous Moon- 
shine, in Representations of groups, B. Allison, G. Cliff (eds), Canad. Math. Soc. 
Conf. Proc. 16 (1995), 141-154. 

[33] V. G. Kac, D. Peterson, Defining relations of certain infinite- dimensional groups, 
Asterisque, hors serie (1985), 165-208. 

[34] V. G. Kac, M. Wakimoto, Modular invariant representations of infinite dimensional 
Lie algebras and superalgebras, Proc. Natl. Acad. Sci. U.S.A. 85 (1988), 4956-4960. 

[35] V. G. Kac, M. Wakimoto, Integrabl highest weight modules over affine superalgebras 
and number theory, in Lie Theory and Geometry, Progress in Math. 123 (1994), 
Birkhauser, Boston, 415-456. 

[36] S.-J. Kang, Root multiplicities of Kac-Moody algebras, Duke Math. J. 74 (1994), 
635-666. 

[37] S.-J. Kang, Generalized Kac-Moody algebras and the modular function j, Math. 
Ann. 298 (1994), 373-384. 

[38] S.-J. Kang, Root multiplicities of graded Lie algebras, in Lie Algebras and Their 
Representations, Contemp. Math. 194 (1996), 161-176. 

[39] S.-J. Kang, Free Lie superalgebras and the generalized Witt formula, RIM-GARC 
preprint 96-49 (1996). 

[40] S.-J. Kang, Graded Lie superalgebras and the superdimension formula, J. Algebra 
204 (1998), 597-655. 

[41] S.-J. Kang, M.-H. Kim, Free Lie algebras, generalized Witt formula, and the denom- 
inator identity, J. Algebra 183 (1996), 560-594. 

[42] S.-J. Kang, M.-H. Kim, Dimension formula for graded Lie algebras and its applica- 
tions, RIM-GARC preprint 96-51 (1996), to appear in Trans. Amer. Math. Soc. 



53 



[43] B. Kostant, Lie algebra cohomology and the generalized Borel-Weil theorem, Ann. 
Math. 74 (1961), 329-387. 

[44] L.-S. Liu, Kostant's formula in Kac-Moody Lie algebras, J. Algebra 149 (1992), 
155-178. 

[45] I. G. Macdonald, Affine root systems and Dedekind's rj-function, Invent. Math. 15 
(1972), 91-143. 

[46] I. G. Macdonald, Symmetric functions and Hall polynomials, Oxford Univ. Press, 
England, 1979. 

[47] M. Miyamoto, A generalization of Borcherds algebra and denominator formula, J. 
Algebra 180 (1996), 631-651. 

[48] R. V. Moody, A new class of Lie algebras, J. Algebra 10 (1968), 211-230. 

[49] S. Naito, The strong Bernstein-Gelfand-Gelfand resolution for generalized Kac- 
Moody algebras I: The existence of the resolution, Publ. RIMS, Kyoto Univ. 29 
(1993), 709-730. 

[50] H. Pop, A generalization of Scheunert's theorem on cocycle twisting of color Lie 
algebras, preprint (1997). 

[51] U. Ray, A character formula for generalized Kac-Moody superalgebras, J. Algebra 
177 (1995), 154-163. 

[52] J. Remmel, The combinatorics of (k,l)-hook Schur functions, Contemporary math- 
ematics 34 (1984), 253-287. 

[53] M. Scheunert, The Theory of Lie Superalgebras, Lecture notes in Math. 716, 
Springer- Verlag, Berlin, 1979. 

[54] M. Scheunert, Generalized Lie algebras, J. Math. Phys. 20 (1979), 712-720. 

[55] H. Weyl, Classical Groups, North-Holland, Amsterdam, 1963. 



54 



